Jul 27 16:17:01.199207 ip-10-0-135-70 systemd[1]: Starting Kubernetes Kubelet... Jul 27 16:17:01.606859 ip-10-0-135-70 kubenswrapper[2550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 16:17:01.606859 ip-10-0-135-70 kubenswrapper[2550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 27 16:17:01.606859 ip-10-0-135-70 kubenswrapper[2550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 16:17:01.606859 ip-10-0-135-70 kubenswrapper[2550]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 27 16:17:01.606859 ip-10-0-135-70 kubenswrapper[2550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 16:17:01.609221 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.609126 2550 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 27 16:17:01.613555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613529 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 16:17:01.613555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613555 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 16:17:01.613555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613559 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 16:17:01.613555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613563 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 16:17:01.613555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613566 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 16:17:01.613703 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613569 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 16:17:01.613703 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613571 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 16:17:01.613703 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613576 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 16:17:01.613703 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613580 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 16:17:01.613703 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613583 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 16:17:01.613703 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613586 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 16:17:01.613703 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613590 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 16:17:01.613703 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613593 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 16:17:01.613703 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613596 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 16:17:01.613703 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613598 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 16:17:01.613703 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613600 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 16:17:01.613703 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613603 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 16:17:01.613703 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613606 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 16:17:01.613703 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613608 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 16:17:01.613703 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613611 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 16:17:01.613703 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613613 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 16:17:01.613703 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613616 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 16:17:01.613703 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613618 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 16:17:01.613703 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613621 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613624 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613626 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613629 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613631 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613634 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613636 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613638 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613641 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613643 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613646 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613648 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613651 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613654 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613656 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613660 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613663 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613666 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613669 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613671 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 16:17:01.614184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613674 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613676 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613678 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613681 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613683 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613685 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613688 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613690 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613692 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613695 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613697 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613700 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613702 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613704 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613707 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613709 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613712 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613715 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613717 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613720 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 16:17:01.614665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613722 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613725 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613728 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613730 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613733 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613736 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613738 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613742 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613745 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613747 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613750 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613753 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613755 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613758 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613760 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613763 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613765 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613768 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613770 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613773 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 16:17:01.615184 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613775 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 16:17:01.615671 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613778 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 16:17:01.615671 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613780 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 16:17:01.615671 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613782 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 16:17:01.615671 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613786 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 16:17:01.615671 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613790 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 16:17:01.615671 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613793 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 16:17:01.615671 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613797 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 16:17:01.615671 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613799 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 16:17:01.615671 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613802 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 16:17:01.615671 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613804 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 16:17:01.615671 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613807 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 16:17:01.615671 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613810 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 16:17:01.615671 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613815 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 16:17:01.615671 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613818 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 16:17:01.615671 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.613821 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 16:17:01.616465 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616452 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 16:17:01.616465 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616466 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616470 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616473 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616476 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616480 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616483 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616486 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616488 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616491 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616494 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616496 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616499 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616501 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616504 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616506 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616510 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616514 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616516 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616519 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616522 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 16:17:01.616551 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616524 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 16:17:01.617059 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616527 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 16:17:01.617059 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616529 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 16:17:01.617059 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616532 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 16:17:01.617059 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616534 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 16:17:01.617059 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616537 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 16:17:01.617059 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616539 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 16:17:01.617059 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616541 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 16:17:01.617059 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616544 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 27 16:17:01.617059 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616546 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 16:17:01.617059 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616549 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 16:17:01.617059 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616551 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 16:17:01.617059 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616554 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 16:17:01.617059 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616556 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 16:17:01.617059 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616559 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 16:17:01.617059 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616562 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 16:17:01.617059 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616565 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 16:17:01.617059 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616568 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 16:17:01.617059 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616570 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 16:17:01.617059 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616572 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 16:17:01.617586 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616575 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 16:17:01.617586 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616577 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 16:17:01.617586 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616579 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 16:17:01.617586 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616582 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 16:17:01.617586 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616584 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 16:17:01.617586 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616587 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 16:17:01.617586 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616589 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 16:17:01.617586 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616592 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 16:17:01.617586 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616594 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 16:17:01.617586 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616596 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 16:17:01.617586 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616599 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 16:17:01.617586 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616602 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 16:17:01.617586 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616604 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 16:17:01.617586 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616608 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 16:17:01.617586 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616611 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 16:17:01.617586 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616615 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 16:17:01.617586 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616619 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 16:17:01.617586 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616622 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 16:17:01.617586 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616624 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616627 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616629 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616632 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616634 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616637 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616639 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616641 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616644 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616646 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616649 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616652 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616654 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616657 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616660 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616662 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616664 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616667 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616671 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616673 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 16:17:01.618064 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616676 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616680 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616684 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616686 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616689 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616692 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616694 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616698 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616701 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616703 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616706 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616709 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616711 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616713 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616716 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616718 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616721 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616723 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616726 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.616728 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 16:17:01.618555 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617124 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617129 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617133 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617135 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617139 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617141 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617144 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617146 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617149 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617152 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617154 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617157 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617160 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617163 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617166 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617168 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617171 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617174 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617177 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617180 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 16:17:01.619036 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617182 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 16:17:01.619546 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617185 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 16:17:01.619546 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617187 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 16:17:01.619546 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617190 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 16:17:01.619546 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617193 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 16:17:01.619546 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617195 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 16:17:01.619546 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617198 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 16:17:01.619546 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617201 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 16:17:01.619546 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617203 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 16:17:01.619546 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617206 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 16:17:01.619546 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617209 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 16:17:01.619546 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617211 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 16:17:01.619546 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617214 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 16:17:01.619546 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617218 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 16:17:01.619546 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617221 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 16:17:01.619546 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617223 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 16:17:01.619546 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617226 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 16:17:01.619546 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617229 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 16:17:01.619546 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617232 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 16:17:01.619546 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617234 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 16:17:01.620031 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617236 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 16:17:01.620031 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617239 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 16:17:01.620031 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617242 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 16:17:01.620031 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617244 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 16:17:01.620031 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617246 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 16:17:01.620031 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617251 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 16:17:01.620031 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617254 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 16:17:01.620031 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617257 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 16:17:01.620031 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617260 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 16:17:01.620031 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617263 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 16:17:01.620031 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617266 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 16:17:01.620031 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617269 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 16:17:01.620031 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617272 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 16:17:01.620031 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617274 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 16:17:01.620031 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617277 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 27 16:17:01.620031 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617280 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 16:17:01.620031 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617282 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 16:17:01.620031 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617285 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 16:17:01.620031 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617287 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617289 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617292 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617294 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617297 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617299 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617302 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617304 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617307 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617310 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617312 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617315 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617318 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617320 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617323 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617325 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617328 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617330 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617332 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617335 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617337 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 16:17:01.620484 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617341 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 16:17:01.621002 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617343 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 16:17:01.621002 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617346 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 16:17:01.621002 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617348 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 16:17:01.621002 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617351 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 16:17:01.621002 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617354 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 16:17:01.621002 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617356 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 16:17:01.621002 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617359 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 16:17:01.621002 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617361 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 16:17:01.621002 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617364 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 16:17:01.621002 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617368 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 16:17:01.621002 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617371 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 16:17:01.621002 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617374 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 16:17:01.621002 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617377 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 16:17:01.621002 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617380 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 16:17:01.621002 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617383 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 16:17:01.621002 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617385 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 16:17:01.621002 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617388 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 16:17:01.621002 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617390 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 16:17:01.621002 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.617393 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617468 2550 flags.go:64] FLAG: --address="0.0.0.0" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617476 2550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617483 2550 flags.go:64] FLAG: --anonymous-auth="true" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617487 2550 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617492 2550 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617495 2550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617500 2550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617504 2550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617507 2550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617510 2550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617513 2550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617516 2550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617520 2550 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617522 2550 flags.go:64] FLAG: --cgroup-root="" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617525 2550 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617528 2550 flags.go:64] FLAG: --client-ca-file="" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617533 2550 flags.go:64] FLAG: --cloud-provider="external" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617536 2550 flags.go:64] FLAG: --cluster-dns="[]" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617540 2550 flags.go:64] FLAG: --cluster-domain="" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617543 2550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617546 2550 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617549 2550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617553 2550 flags.go:64] FLAG: --container-log-max-files="5" Jul 27 16:17:01.621461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617557 2550 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617560 2550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617563 2550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617569 2550 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617572 2550 flags.go:64] FLAG: --contention-profiling="false" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617575 2550 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617578 2550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617581 2550 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617584 2550 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617588 2550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617591 2550 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617594 2550 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617597 2550 flags.go:64] FLAG: --enable-load-reader="false" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617600 2550 flags.go:64] FLAG: --enable-server="true" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617603 2550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617608 2550 flags.go:64] FLAG: --event-burst="100" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617611 2550 flags.go:64] FLAG: --event-qps="50" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617614 2550 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617617 2550 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617619 2550 flags.go:64] FLAG: --eviction-hard="" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617623 2550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617626 2550 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617629 2550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617632 2550 flags.go:64] FLAG: --eviction-soft="" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617634 2550 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 27 16:17:01.622045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617639 2550 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617642 2550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617645 2550 flags.go:64] FLAG: --experimental-mounter-path="" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617648 2550 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617651 2550 flags.go:64] FLAG: --fail-swap-on="true" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617654 2550 flags.go:64] FLAG: --feature-gates="" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617657 2550 flags.go:64] FLAG: --file-check-frequency="20s" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617660 2550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617663 2550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617666 2550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617670 2550 flags.go:64] FLAG: --healthz-port="10248" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617673 2550 flags.go:64] FLAG: --help="false" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617676 2550 flags.go:64] FLAG: --hostname-override="ip-10-0-135-70.ec2.internal" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617679 2550 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617682 2550 flags.go:64] FLAG: --http-check-frequency="20s" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617684 2550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617688 2550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617691 2550 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617694 2550 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617697 2550 flags.go:64] FLAG: --image-service-endpoint="" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617700 2550 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617702 2550 flags.go:64] FLAG: --kube-api-burst="100" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617705 2550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617709 2550 flags.go:64] FLAG: --kube-api-qps="50" Jul 27 16:17:01.622722 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617711 2550 flags.go:64] FLAG: --kube-reserved="" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617714 2550 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617717 2550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617720 2550 flags.go:64] FLAG: --kubelet-cgroups="" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617723 2550 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617725 2550 flags.go:64] FLAG: --lock-file="" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617728 2550 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617731 2550 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617735 2550 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617740 2550 flags.go:64] FLAG: --log-json-split-stream="false" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617743 2550 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617746 2550 flags.go:64] FLAG: --log-text-split-stream="false" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617749 2550 flags.go:64] FLAG: --logging-format="text" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617752 2550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617755 2550 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617758 2550 flags.go:64] FLAG: --manifest-url="" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617761 2550 flags.go:64] FLAG: --manifest-url-header="" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617765 2550 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617769 2550 flags.go:64] FLAG: --max-open-files="1000000" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617773 2550 flags.go:64] FLAG: --max-pods="110" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617776 2550 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617779 2550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617782 2550 flags.go:64] FLAG: --memory-manager-policy="None" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617785 2550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617787 2550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 27 16:17:01.623336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617790 2550 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617793 2550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617800 2550 flags.go:64] FLAG: --node-status-max-images="50" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617803 2550 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617806 2550 flags.go:64] FLAG: --oom-score-adj="-999" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617809 2550 flags.go:64] FLAG: --pod-cidr="" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617813 2550 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617818 2550 flags.go:64] FLAG: --pod-manifest-path="" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617821 2550 flags.go:64] FLAG: --pod-max-pids="-1" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617824 2550 flags.go:64] FLAG: --pods-per-core="0" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617827 2550 flags.go:64] FLAG: --port="10250" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617829 2550 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617832 2550 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-031797fcabd50d577" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617835 2550 flags.go:64] FLAG: --qos-reserved="" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617838 2550 flags.go:64] FLAG: --read-only-port="10255" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617841 2550 flags.go:64] FLAG: --register-node="true" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617845 2550 flags.go:64] FLAG: --register-with-taints="" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617848 2550 flags.go:64] FLAG: --registry-burst="10" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617851 2550 flags.go:64] FLAG: --registry-qps="5" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617854 2550 flags.go:64] FLAG: --reserved-cpus="" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617857 2550 flags.go:64] FLAG: --reserved-memory="" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617860 2550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617863 2550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617866 2550 flags.go:64] FLAG: --rotate-certificates="false" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617869 2550 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 27 16:17:01.623972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617872 2550 flags.go:64] FLAG: --runonce="false" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617875 2550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617878 2550 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617881 2550 flags.go:64] FLAG: --seccomp-default="false" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617884 2550 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617887 2550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617890 2550 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617892 2550 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617895 2550 flags.go:64] FLAG: --storage-driver-password="root" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617911 2550 flags.go:64] FLAG: --storage-driver-secure="false" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617915 2550 flags.go:64] FLAG: --storage-driver-table="stats" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617918 2550 flags.go:64] FLAG: --storage-driver-user="root" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617920 2550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617923 2550 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617930 2550 flags.go:64] FLAG: --system-cgroups="" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617933 2550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617938 2550 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617941 2550 flags.go:64] FLAG: --tls-cert-file="" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617944 2550 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617947 2550 flags.go:64] FLAG: --tls-min-version="" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617950 2550 flags.go:64] FLAG: --tls-private-key-file="" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617953 2550 flags.go:64] FLAG: --topology-manager-policy="none" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617955 2550 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617960 2550 flags.go:64] FLAG: --topology-manager-scope="container" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617963 2550 flags.go:64] FLAG: --v="2" Jul 27 16:17:01.624583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617967 2550 flags.go:64] FLAG: --version="false" Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617971 2550 flags.go:64] FLAG: --vmodule="" Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617975 2550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.617978 2550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618091 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618095 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618098 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618101 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618103 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618106 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618108 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618111 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618113 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618116 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618118 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618121 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618124 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618127 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618129 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618132 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 16:17:01.625207 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618135 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618139 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618144 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618148 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618152 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618157 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618175 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618179 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618182 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618184 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618188 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618191 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618195 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618199 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618202 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618205 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618208 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618210 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618213 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618217 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 16:17:01.625693 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618220 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 16:17:01.626234 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618223 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 16:17:01.626234 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618225 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 16:17:01.626234 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618228 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 16:17:01.626234 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618230 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 16:17:01.626234 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618233 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 16:17:01.626234 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618235 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 16:17:01.626234 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618238 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 16:17:01.626234 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618240 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 16:17:01.626234 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618244 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 16:17:01.626234 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618248 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 16:17:01.626234 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618250 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 16:17:01.626234 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618253 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 16:17:01.626234 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618256 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 16:17:01.626234 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618258 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 16:17:01.626234 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618261 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 16:17:01.626234 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618263 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 16:17:01.626234 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618266 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 16:17:01.626234 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618268 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 16:17:01.626234 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618271 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 16:17:01.626732 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618273 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 16:17:01.626732 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618275 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 16:17:01.626732 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618279 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 16:17:01.626732 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618282 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 16:17:01.626732 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618284 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 16:17:01.626732 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618287 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 27 16:17:01.626732 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618289 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 16:17:01.626732 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618292 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 16:17:01.626732 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618295 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 16:17:01.626732 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618298 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 16:17:01.626732 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618300 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 16:17:01.626732 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618305 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 16:17:01.626732 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618309 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 16:17:01.626732 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618312 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 16:17:01.626732 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618315 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 16:17:01.626732 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618318 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 16:17:01.626732 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618321 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 16:17:01.626732 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618323 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 16:17:01.626732 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618325 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618328 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618330 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618333 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618335 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618338 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618340 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618342 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618345 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618347 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618350 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618353 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618355 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618357 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618360 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618362 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618366 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618368 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618371 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618373 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 16:17:01.627220 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618376 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 16:17:01.627696 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618378 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 16:17:01.627696 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618381 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 16:17:01.627696 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618384 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 16:17:01.627696 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.618386 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 16:17:01.627696 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.619213 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 16:17:01.627696 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.626356 2550 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 27 16:17:01.627696 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.626372 2550 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 27 16:17:01.627696 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.626391 2550 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 27 16:17:01.627696 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.626397 2550 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 27 16:17:01.627696 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626454 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 16:17:01.627696 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626460 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 16:17:01.627696 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626463 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 16:17:01.627696 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626466 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 16:17:01.627696 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626469 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 16:17:01.627696 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626472 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626475 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626478 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626480 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626483 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626487 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626491 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626493 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626496 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626499 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626501 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626504 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626506 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626509 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626511 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626514 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626516 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626519 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626522 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626525 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 16:17:01.628083 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626527 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626530 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626532 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626536 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626539 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626542 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626544 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626547 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626549 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626552 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626554 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626557 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626559 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626561 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626564 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626566 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626569 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626571 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626574 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626576 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 16:17:01.628660 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626579 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626581 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626584 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626587 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626589 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626593 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626596 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626599 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626601 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626604 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626607 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626610 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626613 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626615 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626618 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626621 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626624 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626626 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626629 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626631 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 16:17:01.629457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626634 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 16:17:01.630448 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626636 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 16:17:01.630448 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626639 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 16:17:01.630448 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626641 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 16:17:01.630448 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626643 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 16:17:01.630448 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626646 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 16:17:01.630448 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626648 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 16:17:01.630448 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626651 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 16:17:01.630448 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626653 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 16:17:01.630448 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626656 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 16:17:01.630448 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626658 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 16:17:01.630448 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626660 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 16:17:01.630448 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626663 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 16:17:01.630448 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626666 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 16:17:01.630448 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626668 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 16:17:01.630448 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626670 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 16:17:01.630448 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626673 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 16:17:01.630448 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626675 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 16:17:01.630448 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626678 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 16:17:01.630448 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626680 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 16:17:01.631150 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626682 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 16:17:01.631150 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626686 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 16:17:01.631150 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626688 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 16:17:01.631150 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626691 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 16:17:01.631150 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626694 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 16:17:01.631150 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626696 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 16:17:01.631150 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626699 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 16:17:01.631150 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626702 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 16:17:01.631150 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626704 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 16:17:01.631150 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626707 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 16:17:01.631150 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626711 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 16:17:01.631150 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626714 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 16:17:01.631150 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626717 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 27 16:17:01.631150 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626720 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 16:17:01.631150 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.626725 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 16:17:01.631646 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626842 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 16:17:01.631646 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626847 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 16:17:01.631646 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626850 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 16:17:01.631646 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626853 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 16:17:01.631646 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626856 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 16:17:01.631646 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626859 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 16:17:01.631646 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626861 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 16:17:01.631646 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626864 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 16:17:01.631646 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626866 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 16:17:01.631646 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626869 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 16:17:01.631646 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626871 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 16:17:01.631646 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626874 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 16:17:01.631646 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626877 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 16:17:01.631646 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626879 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 16:17:01.631646 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626882 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 16:17:01.631646 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626884 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 16:17:01.631646 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626887 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 16:17:01.631646 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626889 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 16:17:01.631646 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626891 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626894 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626897 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626916 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626919 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626922 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626925 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626927 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626930 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626932 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626935 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626938 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626940 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626943 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626946 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626949 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626951 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626953 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626956 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626959 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 16:17:01.632167 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626961 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626963 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626966 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626968 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626971 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626973 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626976 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626978 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626981 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626984 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626986 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626988 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626991 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626993 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626996 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.626999 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627001 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627004 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627006 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627009 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 16:17:01.632668 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627011 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627014 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627016 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627019 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627022 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627025 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627028 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627031 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627034 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627036 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627039 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627041 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627044 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627046 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627048 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627051 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627053 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627055 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627058 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627060 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 16:17:01.633186 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627063 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627065 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627068 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627070 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627072 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627075 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627077 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627080 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627083 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627085 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627087 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627090 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627092 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627094 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627097 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627099 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627102 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627106 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627109 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627112 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 16:17:01.633697 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:01.627115 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 16:17:01.634243 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.627120 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 16:17:01.634243 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.627232 2550 server.go:962] "Client rotation is on, will bootstrap in background" Jul 27 16:17:01.634243 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.631504 2550 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 27 16:17:01.634243 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.633179 2550 server.go:1019] "Starting client certificate rotation" Jul 27 16:17:01.634243 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.633285 2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 16:17:01.634243 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.633319 2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 16:17:01.635407 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.635383 2550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 16:17:01.636556 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.636535 2550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 16:17:01.644578 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.644560 2550 log.go:25] "Validated CRI v1 runtime API" Jul 27 16:17:01.646467 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.646451 2550 log.go:25] "Validated CRI v1 image API" Jul 27 16:17:01.648589 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.648569 2550 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 27 16:17:01.651613 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.651595 2550 fs.go:135] Filesystem UUIDs: map[708b1825-611b-41db-9c17-cb46ab69cd54:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 913477b9-cebd-48a6-a75a-d75e6c43ad34:/dev/nvme0n1p3] Jul 27 16:17:01.651680 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.651613 2550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 27 16:17:01.657008 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.656886 2550 manager.go:216] Machine: {Timestamp:2026-07-27 16:17:01.655460867 +0000 UTC m=+0.327995357 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098198 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a2292698e77cc8e4d036144d48ca3 SystemUUID:ec2a2292-698e-77cc-8e4d-036144d48ca3 BootID:2eb6dc52-2513-4933-9c5f-9892975741ee Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d2:98:e9:ce:93 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d2:98:e9:ce:93 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:ba:72:e5:f1:4c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 27 16:17:01.657008 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.657004 2550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 27 16:17:01.657112 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.657079 2550 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 27 16:17:01.660174 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.660147 2550 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 27 16:17:01.660323 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.660178 2550 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-70.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 27 16:17:01.660854 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.660845 2550 topology_manager.go:138] "Creating topology manager with none policy" Jul 27 16:17:01.660889 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.660857 2550 container_manager_linux.go:309] "Creating device plugin manager" Jul 27 16:17:01.660889 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.660865 2550 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 16:17:01.660889 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.660877 2550 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 16:17:01.662060 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.662050 2550 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 27 16:17:01.663877 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.663865 2550 state_mem.go:36] "Initialized new in-memory state store" Jul 27 16:17:01.664004 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.663995 2550 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 27 16:17:01.664048 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.664025 2550 kubelet.go:486] "Attempting to sync node with API server" Jul 27 16:17:01.664048 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.664035 2550 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 27 16:17:01.664048 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.664048 2550 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 27 16:17:01.664157 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.664058 2550 kubelet.go:398] "Adding apiserver pod source" Jul 27 16:17:01.664157 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.664067 2550 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 27 16:17:01.664490 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.664465 2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 16:17:01.664538 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.664492 2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 16:17:01.665303 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.665284 2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 16:17:01.668633 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.668616 2550 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 27 16:17:01.671035 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.671019 2550 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 27 16:17:01.671103 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.671051 2550 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 27 16:17:01.672771 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.672755 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 27 16:17:01.672820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.672781 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 27 16:17:01.672820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.672788 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 27 16:17:01.672820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.672794 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 27 16:17:01.672820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.672802 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 27 16:17:01.672820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.672808 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 27 16:17:01.672820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.672815 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 27 16:17:01.672820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.672820 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 27 16:17:01.673034 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.672829 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 27 16:17:01.673034 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.672838 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 27 16:17:01.673034 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.672846 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 27 16:17:01.673034 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.672851 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 27 16:17:01.674061 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.674048 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 27 16:17:01.674093 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.674064 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 27 16:17:01.677293 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.677280 2550 server.go:1268] "Started kubelet" Jul 27 16:17:01.677934 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.677881 2550 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 27 16:17:01.678054 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.677997 2550 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 27 16:17:01.678130 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.678078 2550 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 27 16:17:01.678310 ip-10-0-135-70 systemd[1]: Started Kubernetes Kubelet. Jul 27 16:17:01.679263 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:01.679155 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-70.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 27 16:17:01.679263 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:01.679159 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 27 16:17:01.679263 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.679214 2550 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-70.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 27 16:17:01.679828 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.679801 2550 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 27 16:17:01.679895 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.679830 2550 server.go:310] "Adding debug handlers to kubelet server" Jul 27 16:17:01.682247 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:01.682223 2550 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 27 16:17:01.682895 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:01.682132 2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-70.ec2.internal.18c631c4591aaaed default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-70.ec2.internal,UID:ip-10-0-135-70.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-70.ec2.internal,},FirstTimestamp:2026-07-27 16:17:01.677251309 +0000 UTC m=+0.349785798,LastTimestamp:2026-07-27 16:17:01.677251309 +0000 UTC m=+0.349785798,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-70.ec2.internal,}" Jul 27 16:17:01.689669 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.689656 2550 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 27 16:17:01.689743 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.689711 2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 27 16:17:01.689794 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:01.689765 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-70.ec2.internal\" not found" Jul 27 16:17:01.690374 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.690357 2550 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 27 16:17:01.690467 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.690379 2550 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 27 16:17:01.690523 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.690467 2550 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 27 16:17:01.690523 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.690484 2550 factory.go:55] Registering systemd factory Jul 27 16:17:01.690523 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.690495 2550 reconstruct.go:97] "Volume reconstruction finished" Jul 27 16:17:01.690523 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.690505 2550 reconciler.go:29] "Reconciler: start to sync state" Jul 27 16:17:01.690523 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.690508 2550 factory.go:223] Registration of the systemd container factory successfully Jul 27 16:17:01.690781 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.690768 2550 factory.go:153] Registering CRI-O factory Jul 27 16:17:01.690833 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.690784 2550 factory.go:223] Registration of the crio container factory successfully Jul 27 16:17:01.690833 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.690824 2550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 27 16:17:01.690894 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.690849 2550 factory.go:103] Registering Raw factory Jul 27 16:17:01.690894 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.690883 2550 manager.go:1196] Started watching for new ooms in manager Jul 27 16:17:01.692171 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.691598 2550 manager.go:349] Starting recovery of all containers Jul 27 16:17:01.692771 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:01.692742 2550 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-70.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 27 16:17:01.692891 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:01.692867 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 27 16:17:01.703435 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.703248 2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-47gfd" Jul 27 16:17:01.703838 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.703819 2550 manager.go:354] Recovery completed Jul 27 16:17:01.709401 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.709386 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 16:17:01.709599 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.709585 2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-47gfd" Jul 27 16:17:01.714054 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.714038 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasSufficientMemory" Jul 27 16:17:01.714124 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.714069 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 16:17:01.714124 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.714083 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasSufficientPID" Jul 27 16:17:01.714615 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.714598 2550 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 27 16:17:01.714615 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.714613 2550 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 27 16:17:01.714719 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.714619 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 16:17:01.714719 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.714628 2550 state_mem.go:36] "Initialized new in-memory state store" Jul 27 16:17:01.716143 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.716128 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasSufficientMemory" Jul 27 16:17:01.716204 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.716153 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 16:17:01.716204 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.716189 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasSufficientPID" Jul 27 16:17:01.716270 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.716227 2550 policy_none.go:49] "None policy: Start" Jul 27 16:17:01.716270 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.716246 2550 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 27 16:17:01.716270 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.716256 2550 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 27 16:17:01.717544 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.717533 2550 policy_none.go:47] "Start" Jul 27 16:17:01.748458 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.748443 2550 manager.go:337] "Starting Device Plugin manager" Jul 27 16:17:01.748548 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:01.748480 2550 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 27 16:17:01.748548 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.748490 2550 server.go:87] "Starting device plugin registration server" Jul 27 16:17:01.748759 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.748746 2550 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 27 16:17:01.748822 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.748761 2550 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 27 16:17:01.748866 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.748843 2550 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 27 16:17:01.748973 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.748946 2550 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 27 16:17:01.748973 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.748958 2550 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 27 16:17:01.749432 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:01.749418 2550 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 27 16:17:01.749505 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:01.749467 2550 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-70.ec2.internal\" not found" Jul 27 16:17:01.841810 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.841778 2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 27 16:17:01.843350 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.843057 2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 27 16:17:01.843350 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.843093 2550 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 27 16:17:01.843350 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.843116 2550 kubelet.go:2448] "Starting kubelet main sync loop" Jul 27 16:17:01.843350 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:01.843147 2550 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 27 16:17:01.845513 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.845496 2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 16:17:01.849435 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.849420 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 16:17:01.850181 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.850169 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasSufficientMemory" Jul 27 16:17:01.850246 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.850198 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 16:17:01.850246 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.850209 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasSufficientPID" Jul 27 16:17:01.850246 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.850232 2550 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-70.ec2.internal" Jul 27 16:17:01.858956 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.858918 2550 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-70.ec2.internal" Jul 27 16:17:01.858956 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:01.858952 2550 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-70.ec2.internal\": node \"ip-10-0-135-70.ec2.internal\" not found" Jul 27 16:17:01.924482 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:01.924450 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-70.ec2.internal\" not found" Jul 27 16:17:01.943555 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.943531 2550 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-70.ec2.internal"] Jul 27 16:17:01.943620 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.943599 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 16:17:01.944541 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.944522 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasSufficientMemory" Jul 27 16:17:01.944599 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.944552 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 16:17:01.944599 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.944562 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasSufficientPID" Jul 27 16:17:01.945862 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.945849 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 16:17:01.946018 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.946003 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal" Jul 27 16:17:01.946084 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.946076 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 16:17:01.946467 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.946450 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasSufficientMemory" Jul 27 16:17:01.946553 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.946483 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 16:17:01.946553 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.946497 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasSufficientPID" Jul 27 16:17:01.946635 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.946561 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasSufficientMemory" Jul 27 16:17:01.946635 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.946588 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 16:17:01.946635 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.946599 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasSufficientPID" Jul 27 16:17:01.947968 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.947952 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.ec2.internal" Jul 27 16:17:01.948016 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.947987 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 16:17:01.948624 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.948608 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasSufficientMemory" Jul 27 16:17:01.948693 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.948635 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 16:17:01.948693 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:01.948645 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeHasSufficientPID" Jul 27 16:17:01.982620 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:01.982600 2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-70.ec2.internal\" not found" node="ip-10-0-135-70.ec2.internal" Jul 27 16:17:01.986995 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:01.986977 2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-70.ec2.internal\" not found" node="ip-10-0-135-70.ec2.internal" Jul 27 16:17:02.024677 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:02.024655 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-70.ec2.internal\" not found" Jul 27 16:17:02.091171 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.091143 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/efedf659439c0f4b07ff2be7c9f4e1bd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-70.ec2.internal\" (UID: \"efedf659439c0f4b07ff2be7c9f4e1bd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.ec2.internal" Jul 27 16:17:02.091258 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.091172 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/158785e5c4cf0d4aef5b4d9ab831c0ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal\" (UID: \"158785e5c4cf0d4aef5b4d9ab831c0ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal" Jul 27 16:17:02.091258 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.091193 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/158785e5c4cf0d4aef5b4d9ab831c0ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal\" (UID: \"158785e5c4cf0d4aef5b4d9ab831c0ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal" Jul 27 16:17:02.125254 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:02.125193 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-70.ec2.internal\" not found" Jul 27 16:17:02.192071 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.192043 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/158785e5c4cf0d4aef5b4d9ab831c0ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal\" (UID: \"158785e5c4cf0d4aef5b4d9ab831c0ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal" Jul 27 16:17:02.192157 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.192074 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/158785e5c4cf0d4aef5b4d9ab831c0ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal\" (UID: \"158785e5c4cf0d4aef5b4d9ab831c0ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal" Jul 27 16:17:02.192157 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.192100 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/efedf659439c0f4b07ff2be7c9f4e1bd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-70.ec2.internal\" (UID: \"efedf659439c0f4b07ff2be7c9f4e1bd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.ec2.internal" Jul 27 16:17:02.192157 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.192131 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/158785e5c4cf0d4aef5b4d9ab831c0ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal\" (UID: \"158785e5c4cf0d4aef5b4d9ab831c0ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal" Jul 27 16:17:02.192157 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.192138 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/158785e5c4cf0d4aef5b4d9ab831c0ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal\" (UID: \"158785e5c4cf0d4aef5b4d9ab831c0ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal" Jul 27 16:17:02.192279 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.192171 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/efedf659439c0f4b07ff2be7c9f4e1bd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-70.ec2.internal\" (UID: \"efedf659439c0f4b07ff2be7c9f4e1bd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.ec2.internal" Jul 27 16:17:02.226104 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:02.226080 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-70.ec2.internal\" not found" Jul 27 16:17:02.286305 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.286268 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal" Jul 27 16:17:02.291894 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.291521 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.ec2.internal" Jul 27 16:17:02.327164 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:02.327134 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-70.ec2.internal\" not found" Jul 27 16:17:02.427824 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:02.427737 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-70.ec2.internal\" not found" Jul 27 16:17:02.528313 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:02.528278 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-70.ec2.internal\" not found" Jul 27 16:17:02.586206 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.586176 2550 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 16:17:02.628613 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:02.628580 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-70.ec2.internal\" not found" Jul 27 16:17:02.632745 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.632727 2550 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 27 16:17:02.632889 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.632870 2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 16:17:02.632963 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.632896 2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 16:17:02.679804 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.679726 2550 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 16:17:02.715497 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.715442 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 16:12:02 +0000 UTC" deadline="2028-04-14 06:57:13.33718154 +0000 UTC" Jul 27 16:17:02.715497 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.715491 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15038h40m10.621693719s" Jul 27 16:17:02.729621 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:02.729585 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-70.ec2.internal\" not found" Jul 27 16:17:02.794689 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.794520 2550 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 16:17:02.842954 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:02.842925 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158785e5c4cf0d4aef5b4d9ab831c0ca.slice/crio-f2b2497baccf9f5ae37ffa9694c78b4a3f1d6fe924a91a4536cf6901a6293cf4 WatchSource:0}: Error finding container f2b2497baccf9f5ae37ffa9694c78b4a3f1d6fe924a91a4536cf6901a6293cf4: Status 404 returned error can't find the container with id f2b2497baccf9f5ae37ffa9694c78b4a3f1d6fe924a91a4536cf6901a6293cf4 Jul 27 16:17:02.847436 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.847419 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 16:17:02.890290 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.890265 2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal" Jul 27 16:17:02.909801 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.909779 2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 16:17:02.910878 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.910863 2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.ec2.internal" Jul 27 16:17:02.925201 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:02.925183 2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 16:17:03.665896 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.665867 2550 apiserver.go:52] "Watching apiserver" Jul 27 16:17:03.677758 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.677727 2550 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 27 16:17:03.678120 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.678098 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-mg2dr","openshift-network-operator/iptables-alerter-dkzxx","openshift-ovn-kubernetes/ovnkube-node-f9gll","kube-system/konnectivity-agent-tzvpv","kube-system/kube-apiserver-proxy-ip-10-0-135-70.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx","openshift-cluster-node-tuning-operator/tuned-4mx8h","openshift-image-registry/node-ca-tnb6j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal","openshift-multus/multus-9lrvm","openshift-network-diagnostics/network-check-target-26l7z","kube-system/global-pull-secret-syncer-lkclq","openshift-dns/node-resolver-9fnfd","openshift-multus/multus-additional-cni-plugins-q9n8r"] Jul 27 16:17:03.680562 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.680541 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.682069 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.682051 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:03.682182 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:03.682123 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:03.683368 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.683347 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.683676 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.683649 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 27 16:17:03.683773 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.683733 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bfqjr\"" Jul 27 16:17:03.683773 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.683742 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 27 16:17:03.684589 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.684569 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tzvpv" Jul 27 16:17:03.685907 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.685889 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tnb6j" Jul 27 16:17:03.687353 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.687335 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.688100 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.688080 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 27 16:17:03.688214 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.688193 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 27 16:17:03.688440 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.688394 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 27 16:17:03.688533 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.688447 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 27 16:17:03.688734 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.688714 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 27 16:17:03.688829 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.688716 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:03.688829 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:03.688807 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:03.689644 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.689624 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 27 16:17:03.690035 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.690012 2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 27 16:17:03.690163 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.690059 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 27 16:17:03.690289 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.690272 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nnr29\"" Jul 27 16:17:03.690354 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.690293 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8w8np\"" Jul 27 16:17:03.690354 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.690327 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9fnfd" Jul 27 16:17:03.690496 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.690351 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 27 16:17:03.690592 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.690569 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 27 16:17:03.690694 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.690646 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 27 16:17:03.690694 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.690648 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tvrb2\"" Jul 27 16:17:03.690874 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.690856 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 27 16:17:03.691033 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.690882 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 27 16:17:03.691097 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.691077 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5zjsh\"" Jul 27 16:17:03.691150 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.691130 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 27 16:17:03.691645 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.691628 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dkzxx" Jul 27 16:17:03.693002 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.692961 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.694017 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.693996 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7jc59\"" Jul 27 16:17:03.694140 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.693998 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 27 16:17:03.694295 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.694275 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:03.694371 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.694322 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 27 16:17:03.694371 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:03.694354 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:03.695365 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.695327 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 27 16:17:03.695462 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.695373 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5ftkm\"" Jul 27 16:17:03.695462 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.695384 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 27 16:17:03.695462 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.695396 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 27 16:17:03.695618 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.695538 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 27 16:17:03.695789 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.695770 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.696062 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.695963 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-v4tdq\"" Jul 27 16:17:03.698450 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.698435 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 27 16:17:03.698795 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.698774 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-etc-modprobe-d\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.698922 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.698802 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-lib-modules\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.698922 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.698819 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26550e07-19b7-4937-9964-58c93feb47a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.698922 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.698838 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/26550e07-19b7-4937-9964-58c93feb47a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.698922 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.698874 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1f1cd14-e645-4eb1-bef3-22089007cf7f-registration-dir\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.698922 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.698893 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnpnq\" (UniqueName: \"kubernetes.io/projected/f1f1cd14-e645-4eb1-bef3-22089007cf7f-kube-api-access-lnpnq\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.699164 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.698949 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b8bc6ae-9d34-41e4-8079-9149a29e6627-host\") pod \"node-ca-tnb6j\" (UID: \"9b8bc6ae-9d34-41e4-8079-9149a29e6627\") " pod="openshift-image-registry/node-ca-tnb6j" Jul 27 16:17:03.699164 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.698981 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-sys\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.699164 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699008 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca803c9f-3708-4539-be6c-abef843d77d0-etc-tuned\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.699164 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699048 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s74mc\" (UniqueName: \"kubernetes.io/projected/ca803c9f-3708-4539-be6c-abef843d77d0-kube-api-access-s74mc\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.699164 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699087 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26550e07-19b7-4937-9964-58c93feb47a7-system-cni-dir\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.699164 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699131 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-etc-sysctl-d\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.699470 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699185 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5445bc4e-d01a-48c8-8c71-6541ad9a487e-konnectivity-ca\") pod \"konnectivity-agent-tzvpv\" (UID: \"5445bc4e-d01a-48c8-8c71-6541ad9a487e\") " pod="kube-system/konnectivity-agent-tzvpv" Jul 27 16:17:03.699470 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699217 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1f1cd14-e645-4eb1-bef3-22089007cf7f-socket-dir\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.699470 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699242 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1f1cd14-e645-4eb1-bef3-22089007cf7f-device-dir\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.699470 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699270 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b765f995-1ea5-4a72-b3ad-c661cd537410-host-slash\") pod \"iptables-alerter-dkzxx\" (UID: \"b765f995-1ea5-4a72-b3ad-c661cd537410\") " pod="openshift-network-operator/iptables-alerter-dkzxx" Jul 27 16:17:03.699470 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699294 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-host\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.699470 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699321 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/26550e07-19b7-4937-9964-58c93feb47a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.699470 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699348 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5445bc4e-d01a-48c8-8c71-6541ad9a487e-agent-certs\") pod \"konnectivity-agent-tzvpv\" (UID: \"5445bc4e-d01a-48c8-8c71-6541ad9a487e\") " pod="kube-system/konnectivity-agent-tzvpv" Jul 27 16:17:03.699470 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699373 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92a27900-e3d3-44c2-a7da-7c3ff58b7cfc-hosts-file\") pod \"node-resolver-9fnfd\" (UID: \"92a27900-e3d3-44c2-a7da-7c3ff58b7cfc\") " pod="openshift-dns/node-resolver-9fnfd" Jul 27 16:17:03.699470 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699398 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-etc-sysconfig\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.699470 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699441 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-var-lib-kubelet\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.700035 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699474 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26550e07-19b7-4937-9964-58c93feb47a7-cnibin\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.700035 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699507 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/26550e07-19b7-4937-9964-58c93feb47a7-os-release\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.700035 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699553 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1f1cd14-e645-4eb1-bef3-22089007cf7f-etc-selinux\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.700035 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699575 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a27900-e3d3-44c2-a7da-7c3ff58b7cfc-tmp-dir\") pod \"node-resolver-9fnfd\" (UID: \"92a27900-e3d3-44c2-a7da-7c3ff58b7cfc\") " pod="openshift-dns/node-resolver-9fnfd" Jul 27 16:17:03.700035 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699602 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c98l6\" (UniqueName: \"kubernetes.io/projected/b765f995-1ea5-4a72-b3ad-c661cd537410-kube-api-access-c98l6\") pod \"iptables-alerter-dkzxx\" (UID: \"b765f995-1ea5-4a72-b3ad-c661cd537410\") " pod="openshift-network-operator/iptables-alerter-dkzxx" Jul 27 16:17:03.700035 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.699797 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-etc-sysctl-conf\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.700035 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.700026 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-etc-systemd\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.700414 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.700073 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-run\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.700414 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.700108 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca803c9f-3708-4539-be6c-abef843d77d0-tmp\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.700414 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.700146 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1f1cd14-e645-4eb1-bef3-22089007cf7f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.700414 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.700221 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8jqc\" (UniqueName: \"kubernetes.io/projected/9b8bc6ae-9d34-41e4-8079-9149a29e6627-kube-api-access-m8jqc\") pod \"node-ca-tnb6j\" (UID: \"9b8bc6ae-9d34-41e4-8079-9149a29e6627\") " pod="openshift-image-registry/node-ca-tnb6j" Jul 27 16:17:03.700414 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.700254 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b765f995-1ea5-4a72-b3ad-c661cd537410-iptables-alerter-script\") pod \"iptables-alerter-dkzxx\" (UID: \"b765f995-1ea5-4a72-b3ad-c661cd537410\") " pod="openshift-network-operator/iptables-alerter-dkzxx" Jul 27 16:17:03.700414 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.700265 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h57m8\"" Jul 27 16:17:03.700414 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.700289 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-etc-kubernetes\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.700414 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.700325 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1f1cd14-e645-4eb1-bef3-22089007cf7f-sys-fs\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.700414 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.700372 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs\") pod \"network-metrics-daemon-mg2dr\" (UID: \"fc3e4156-6cfc-430a-8842-a9d801cfb9f0\") " pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:03.700414 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.700415 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x648c\" (UniqueName: \"kubernetes.io/projected/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-kube-api-access-x648c\") pod \"network-metrics-daemon-mg2dr\" (UID: \"fc3e4156-6cfc-430a-8842-a9d801cfb9f0\") " pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:03.700939 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.700445 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26550e07-19b7-4937-9964-58c93feb47a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.700939 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.700481 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7bxt\" (UniqueName: \"kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt\") pod \"network-check-target-26l7z\" (UID: \"b1c1e78a-1087-4293-aca2-a34e14ee090b\") " pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:03.700939 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.700597 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 27 16:17:03.700939 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.700623 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b8bc6ae-9d34-41e4-8079-9149a29e6627-serviceca\") pod \"node-ca-tnb6j\" (UID: \"9b8bc6ae-9d34-41e4-8079-9149a29e6627\") " pod="openshift-image-registry/node-ca-tnb6j" Jul 27 16:17:03.700939 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.700672 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt5rv\" (UniqueName: \"kubernetes.io/projected/92a27900-e3d3-44c2-a7da-7c3ff58b7cfc-kube-api-access-dt5rv\") pod \"node-resolver-9fnfd\" (UID: \"92a27900-e3d3-44c2-a7da-7c3ff58b7cfc\") " pod="openshift-dns/node-resolver-9fnfd" Jul 27 16:17:03.700939 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.700762 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrzc8\" (UniqueName: \"kubernetes.io/projected/26550e07-19b7-4937-9964-58c93feb47a7-kube-api-access-nrzc8\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.700939 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.700862 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 27 16:17:03.701292 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.700960 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 27 16:17:03.701353 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.701301 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 27 16:17:03.701478 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.701458 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 27 16:17:03.709873 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.709847 2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 16:17:03.739367 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.739343 2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wpszk" Jul 27 16:17:03.749257 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.749236 2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wpszk" Jul 27 16:17:03.792533 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.792504 2550 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 27 16:17:03.801910 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.801884 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/26550e07-19b7-4937-9964-58c93feb47a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.802043 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.801939 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-cnibin\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.802043 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.801970 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-host-run-k8s-cni-cncf-io\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.802043 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.801997 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92a27900-e3d3-44c2-a7da-7c3ff58b7cfc-hosts-file\") pod \"node-resolver-9fnfd\" (UID: \"92a27900-e3d3-44c2-a7da-7c3ff58b7cfc\") " pod="openshift-dns/node-resolver-9fnfd" Jul 27 16:17:03.802043 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802013 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-etc-sysconfig\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.802260 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802075 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26550e07-19b7-4937-9964-58c93feb47a7-cnibin\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.802260 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802085 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-etc-sysconfig\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.802260 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802092 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92a27900-e3d3-44c2-a7da-7c3ff58b7cfc-hosts-file\") pod \"node-resolver-9fnfd\" (UID: \"92a27900-e3d3-44c2-a7da-7c3ff58b7cfc\") " pod="openshift-dns/node-resolver-9fnfd" Jul 27 16:17:03.802260 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802140 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26550e07-19b7-4937-9964-58c93feb47a7-cnibin\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.802260 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802114 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-log-socket\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.802260 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802178 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-ovn-node-metrics-cert\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.802260 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802199 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1f1cd14-e645-4eb1-bef3-22089007cf7f-etc-selinux\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.802260 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802222 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a27900-e3d3-44c2-a7da-7c3ff58b7cfc-tmp-dir\") pod \"node-resolver-9fnfd\" (UID: \"92a27900-e3d3-44c2-a7da-7c3ff58b7cfc\") " pod="openshift-dns/node-resolver-9fnfd" Jul 27 16:17:03.802621 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802323 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-etc-sysctl-conf\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.802621 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802344 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1f1cd14-e645-4eb1-bef3-22089007cf7f-etc-selinux\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.802621 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802360 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca803c9f-3708-4539-be6c-abef843d77d0-tmp\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.802621 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802388 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-multus-cni-dir\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.802621 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802413 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-host-var-lib-kubelet\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.802621 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802437 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-kubelet-config\") pod \"global-pull-secret-syncer-lkclq\" (UID: \"efd59b32-c96c-4808-9a83-b24cdc2e3dc7\") " pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:03.802621 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802465 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a27900-e3d3-44c2-a7da-7c3ff58b7cfc-tmp-dir\") pod \"node-resolver-9fnfd\" (UID: \"92a27900-e3d3-44c2-a7da-7c3ff58b7cfc\") " pod="openshift-dns/node-resolver-9fnfd" Jul 27 16:17:03.802621 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802472 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1f1cd14-e645-4eb1-bef3-22089007cf7f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.802621 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802481 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-etc-sysctl-conf\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.802621 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802501 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8jqc\" (UniqueName: \"kubernetes.io/projected/9b8bc6ae-9d34-41e4-8079-9149a29e6627-kube-api-access-m8jqc\") pod \"node-ca-tnb6j\" (UID: \"9b8bc6ae-9d34-41e4-8079-9149a29e6627\") " pod="openshift-image-registry/node-ca-tnb6j" Jul 27 16:17:03.802621 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802529 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-host-var-lib-cni-multus\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.802621 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802566 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-etc-kubernetes\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.802621 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802569 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1f1cd14-e645-4eb1-bef3-22089007cf7f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.803361 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802634 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/26550e07-19b7-4937-9964-58c93feb47a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.803361 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802692 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-dbus\") pod \"global-pull-secret-syncer-lkclq\" (UID: \"efd59b32-c96c-4808-9a83-b24cdc2e3dc7\") " pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:03.803361 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802735 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-systemd-units\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.803361 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802767 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1f1cd14-e645-4eb1-bef3-22089007cf7f-sys-fs\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.803361 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802796 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7bxt\" (UniqueName: \"kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt\") pod \"network-check-target-26l7z\" (UID: \"b1c1e78a-1087-4293-aca2-a34e14ee090b\") " pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:03.803361 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802825 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b8bc6ae-9d34-41e4-8079-9149a29e6627-serviceca\") pod \"node-ca-tnb6j\" (UID: \"9b8bc6ae-9d34-41e4-8079-9149a29e6627\") " pod="openshift-image-registry/node-ca-tnb6j" Jul 27 16:17:03.803361 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802864 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1f1cd14-e645-4eb1-bef3-22089007cf7f-sys-fs\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.803361 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802865 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrzc8\" (UniqueName: \"kubernetes.io/projected/26550e07-19b7-4937-9964-58c93feb47a7-kube-api-access-nrzc8\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.803361 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802881 2550 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 27 16:17:03.803361 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802916 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-run-netns\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.803361 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802944 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-ovnkube-config\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.803361 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.802992 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-ovnkube-script-lib\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.803361 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803026 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-etc-modprobe-d\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.803361 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803070 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-lib-modules\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.803361 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803098 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26550e07-19b7-4937-9964-58c93feb47a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.803361 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803143 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-etc-modprobe-d\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.803361 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803126 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/26550e07-19b7-4937-9964-58c93feb47a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.804187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803193 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-kubelet\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.804187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803203 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-lib-modules\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.804187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803225 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96dqc\" (UniqueName: \"kubernetes.io/projected/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-kube-api-access-96dqc\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.804187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803265 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1f1cd14-e645-4eb1-bef3-22089007cf7f-registration-dir\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.804187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803286 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b8bc6ae-9d34-41e4-8079-9149a29e6627-serviceca\") pod \"node-ca-tnb6j\" (UID: \"9b8bc6ae-9d34-41e4-8079-9149a29e6627\") " pod="openshift-image-registry/node-ca-tnb6j" Jul 27 16:17:03.804187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803326 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1f1cd14-e645-4eb1-bef3-22089007cf7f-registration-dir\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.804187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803334 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnpnq\" (UniqueName: \"kubernetes.io/projected/f1f1cd14-e645-4eb1-bef3-22089007cf7f-kube-api-access-lnpnq\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.804187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803371 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-multus-socket-dir-parent\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.804187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803488 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-multus-conf-dir\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.804187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803547 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5445bc4e-d01a-48c8-8c71-6541ad9a487e-konnectivity-ca\") pod \"konnectivity-agent-tzvpv\" (UID: \"5445bc4e-d01a-48c8-8c71-6541ad9a487e\") " pod="kube-system/konnectivity-agent-tzvpv" Jul 27 16:17:03.804187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803570 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1f1cd14-e645-4eb1-bef3-22089007cf7f-socket-dir\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.804187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803593 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1f1cd14-e645-4eb1-bef3-22089007cf7f-device-dir\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.804187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803614 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b765f995-1ea5-4a72-b3ad-c661cd537410-host-slash\") pod \"iptables-alerter-dkzxx\" (UID: \"b765f995-1ea5-4a72-b3ad-c661cd537410\") " pod="openshift-network-operator/iptables-alerter-dkzxx" Jul 27 16:17:03.804187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803634 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-os-release\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.804187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803651 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26550e07-19b7-4937-9964-58c93feb47a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.804187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803655 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-run-systemd\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.804187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803697 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/26550e07-19b7-4937-9964-58c93feb47a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.805073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803707 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1f1cd14-e645-4eb1-bef3-22089007cf7f-socket-dir\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.805073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803699 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-var-lib-openvswitch\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.805073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803724 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1f1cd14-e645-4eb1-bef3-22089007cf7f-device-dir\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.805073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803754 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b765f995-1ea5-4a72-b3ad-c661cd537410-host-slash\") pod \"iptables-alerter-dkzxx\" (UID: \"b765f995-1ea5-4a72-b3ad-c661cd537410\") " pod="openshift-network-operator/iptables-alerter-dkzxx" Jul 27 16:17:03.805073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803756 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5445bc4e-d01a-48c8-8c71-6541ad9a487e-agent-certs\") pod \"konnectivity-agent-tzvpv\" (UID: \"5445bc4e-d01a-48c8-8c71-6541ad9a487e\") " pod="kube-system/konnectivity-agent-tzvpv" Jul 27 16:17:03.805073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803795 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-var-lib-kubelet\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.805073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803822 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/26550e07-19b7-4937-9964-58c93feb47a7-os-release\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.805073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803851 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b57811b-9cd7-40e0-b4c4-3b24a5822001-cni-binary-copy\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.805073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803870 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-var-lib-kubelet\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.805073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803876 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-slash\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.805073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.803934 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-node-log\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.805073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804005 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/26550e07-19b7-4937-9964-58c93feb47a7-os-release\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.805073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804034 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c98l6\" (UniqueName: \"kubernetes.io/projected/b765f995-1ea5-4a72-b3ad-c661cd537410-kube-api-access-c98l6\") pod \"iptables-alerter-dkzxx\" (UID: \"b765f995-1ea5-4a72-b3ad-c661cd537410\") " pod="openshift-network-operator/iptables-alerter-dkzxx" Jul 27 16:17:03.805073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804062 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-etc-systemd\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.805073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804090 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-run\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.805073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804112 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5445bc4e-d01a-48c8-8c71-6541ad9a487e-konnectivity-ca\") pod \"konnectivity-agent-tzvpv\" (UID: \"5445bc4e-d01a-48c8-8c71-6541ad9a487e\") " pod="kube-system/konnectivity-agent-tzvpv" Jul 27 16:17:03.805073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804150 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-run\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.805926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804151 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-etc-systemd\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.805926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804172 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-hostroot\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.805926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804191 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-env-overrides\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.805926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804219 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b765f995-1ea5-4a72-b3ad-c661cd537410-iptables-alerter-script\") pod \"iptables-alerter-dkzxx\" (UID: \"b765f995-1ea5-4a72-b3ad-c661cd537410\") " pod="openshift-network-operator/iptables-alerter-dkzxx" Jul 27 16:17:03.805926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804244 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-etc-kubernetes\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.805926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804278 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret\") pod \"global-pull-secret-syncer-lkclq\" (UID: \"efd59b32-c96c-4808-9a83-b24cdc2e3dc7\") " pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:03.805926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804303 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-run-ovn-kubernetes\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.805926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804338 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-etc-kubernetes\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.805926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804336 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.805926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804389 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs\") pod \"network-metrics-daemon-mg2dr\" (UID: \"fc3e4156-6cfc-430a-8842-a9d801cfb9f0\") " pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:03.805926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804416 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x648c\" (UniqueName: \"kubernetes.io/projected/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-kube-api-access-x648c\") pod \"network-metrics-daemon-mg2dr\" (UID: \"fc3e4156-6cfc-430a-8842-a9d801cfb9f0\") " pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:03.805926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804454 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26550e07-19b7-4937-9964-58c93feb47a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.805926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804478 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-system-cni-dir\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.805926 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:03.804495 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:17:03.805926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804503 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-host-run-multus-certs\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.805926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804529 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-etc-openvswitch\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.805926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804558 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-run-openvswitch\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.806878 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804592 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt5rv\" (UniqueName: \"kubernetes.io/projected/92a27900-e3d3-44c2-a7da-7c3ff58b7cfc-kube-api-access-dt5rv\") pod \"node-resolver-9fnfd\" (UID: \"92a27900-e3d3-44c2-a7da-7c3ff58b7cfc\") " pod="openshift-dns/node-resolver-9fnfd" Jul 27 16:17:03.806878 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804625 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-host-run-netns\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.806878 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804641 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26550e07-19b7-4937-9964-58c93feb47a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.806878 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:03.804673 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs podName:fc3e4156-6cfc-430a-8842-a9d801cfb9f0 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:04.304650093 +0000 UTC m=+2.977184587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs") pod "network-metrics-daemon-mg2dr" (UID: "fc3e4156-6cfc-430a-8842-a9d801cfb9f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:17:03.806878 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804701 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk6vj\" (UniqueName: \"kubernetes.io/projected/8b57811b-9cd7-40e0-b4c4-3b24a5822001-kube-api-access-vk6vj\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.806878 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804724 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b765f995-1ea5-4a72-b3ad-c661cd537410-iptables-alerter-script\") pod \"iptables-alerter-dkzxx\" (UID: \"b765f995-1ea5-4a72-b3ad-c661cd537410\") " pod="openshift-network-operator/iptables-alerter-dkzxx" Jul 27 16:17:03.806878 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804820 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-run-ovn\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.806878 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804875 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-cni-bin\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.806878 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804911 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b57811b-9cd7-40e0-b4c4-3b24a5822001-multus-daemon-config\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.806878 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804954 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-cni-netd\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.806878 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804975 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b8bc6ae-9d34-41e4-8079-9149a29e6627-host\") pod \"node-ca-tnb6j\" (UID: \"9b8bc6ae-9d34-41e4-8079-9149a29e6627\") " pod="openshift-image-registry/node-ca-tnb6j" Jul 27 16:17:03.806878 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.804991 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-sys\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.806878 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.805027 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca803c9f-3708-4539-be6c-abef843d77d0-etc-tuned\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.806878 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.805063 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s74mc\" (UniqueName: \"kubernetes.io/projected/ca803c9f-3708-4539-be6c-abef843d77d0-kube-api-access-s74mc\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.806878 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.805088 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26550e07-19b7-4937-9964-58c93feb47a7-system-cni-dir\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.806878 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.805090 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b8bc6ae-9d34-41e4-8079-9149a29e6627-host\") pod \"node-ca-tnb6j\" (UID: \"9b8bc6ae-9d34-41e4-8079-9149a29e6627\") " pod="openshift-image-registry/node-ca-tnb6j" Jul 27 16:17:03.807855 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.805129 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-host-var-lib-cni-bin\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.807855 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.805142 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-sys\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.807855 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.805156 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-etc-sysctl-d\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.807855 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.805211 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26550e07-19b7-4937-9964-58c93feb47a7-system-cni-dir\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.807855 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.805262 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-etc-sysctl-d\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.807855 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.805293 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-host\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.807855 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.805568 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca803c9f-3708-4539-be6c-abef843d77d0-host\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.807855 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.806723 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca803c9f-3708-4539-be6c-abef843d77d0-tmp\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.807855 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.807424 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca803c9f-3708-4539-be6c-abef843d77d0-etc-tuned\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.807855 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.807557 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5445bc4e-d01a-48c8-8c71-6541ad9a487e-agent-certs\") pod \"konnectivity-agent-tzvpv\" (UID: \"5445bc4e-d01a-48c8-8c71-6541ad9a487e\") " pod="kube-system/konnectivity-agent-tzvpv" Jul 27 16:17:03.810245 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:03.810224 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 16:17:03.810245 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:03.810253 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 16:17:03.810523 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:03.810270 2550 projected.go:196] Error preparing data for projected volume kube-api-access-s7bxt for pod openshift-network-diagnostics/network-check-target-26l7z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:17:03.810523 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:03.810354 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt podName:b1c1e78a-1087-4293-aca2-a34e14ee090b nodeName:}" failed. No retries permitted until 2026-07-27 16:17:04.310334511 +0000 UTC m=+2.982868994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s7bxt" (UniqueName: "kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt") pod "network-check-target-26l7z" (UID: "b1c1e78a-1087-4293-aca2-a34e14ee090b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:17:03.812498 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.812461 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8jqc\" (UniqueName: \"kubernetes.io/projected/9b8bc6ae-9d34-41e4-8079-9149a29e6627-kube-api-access-m8jqc\") pod \"node-ca-tnb6j\" (UID: \"9b8bc6ae-9d34-41e4-8079-9149a29e6627\") " pod="openshift-image-registry/node-ca-tnb6j" Jul 27 16:17:03.813639 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.813607 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrzc8\" (UniqueName: \"kubernetes.io/projected/26550e07-19b7-4937-9964-58c93feb47a7-kube-api-access-nrzc8\") pod \"multus-additional-cni-plugins-q9n8r\" (UID: \"26550e07-19b7-4937-9964-58c93feb47a7\") " pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:03.814413 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.814361 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnpnq\" (UniqueName: \"kubernetes.io/projected/f1f1cd14-e645-4eb1-bef3-22089007cf7f-kube-api-access-lnpnq\") pod \"aws-ebs-csi-driver-node-h69bx\" (UID: \"f1f1cd14-e645-4eb1-bef3-22089007cf7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:03.815373 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.815338 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c98l6\" (UniqueName: \"kubernetes.io/projected/b765f995-1ea5-4a72-b3ad-c661cd537410-kube-api-access-c98l6\") pod \"iptables-alerter-dkzxx\" (UID: \"b765f995-1ea5-4a72-b3ad-c661cd537410\") " pod="openshift-network-operator/iptables-alerter-dkzxx" Jul 27 16:17:03.815571 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.815545 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt5rv\" (UniqueName: \"kubernetes.io/projected/92a27900-e3d3-44c2-a7da-7c3ff58b7cfc-kube-api-access-dt5rv\") pod \"node-resolver-9fnfd\" (UID: \"92a27900-e3d3-44c2-a7da-7c3ff58b7cfc\") " pod="openshift-dns/node-resolver-9fnfd" Jul 27 16:17:03.815942 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.815912 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x648c\" (UniqueName: \"kubernetes.io/projected/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-kube-api-access-x648c\") pod \"network-metrics-daemon-mg2dr\" (UID: \"fc3e4156-6cfc-430a-8842-a9d801cfb9f0\") " pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:03.818642 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.818616 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s74mc\" (UniqueName: \"kubernetes.io/projected/ca803c9f-3708-4539-be6c-abef843d77d0-kube-api-access-s74mc\") pod \"tuned-4mx8h\" (UID: \"ca803c9f-3708-4539-be6c-abef843d77d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.848475 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.848378 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal" event={"ID":"158785e5c4cf0d4aef5b4d9ab831c0ca","Type":"ContainerStarted","Data":"f2b2497baccf9f5ae37ffa9694c78b4a3f1d6fe924a91a4536cf6901a6293cf4"} Jul 27 16:17:03.906076 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906038 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-host-var-lib-cni-bin\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.906272 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906088 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-cnibin\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.906272 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906154 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-host-var-lib-cni-bin\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.906272 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906168 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-cnibin\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.906272 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906197 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-host-run-k8s-cni-cncf-io\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.906272 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906227 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-log-socket\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.906272 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906254 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-ovn-node-metrics-cert\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.906593 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906284 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-multus-cni-dir\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.906593 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906302 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-log-socket\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.906593 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906310 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-host-var-lib-kubelet\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.906593 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906307 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-host-run-k8s-cni-cncf-io\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.906593 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906338 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-kubelet-config\") pod \"global-pull-secret-syncer-lkclq\" (UID: \"efd59b32-c96c-4808-9a83-b24cdc2e3dc7\") " pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:03.906593 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906368 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-host-var-lib-cni-multus\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.906593 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906380 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-host-var-lib-kubelet\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.906593 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906392 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-etc-kubernetes\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.906593 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906417 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-dbus\") pod \"global-pull-secret-syncer-lkclq\" (UID: \"efd59b32-c96c-4808-9a83-b24cdc2e3dc7\") " pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:03.906593 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906440 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-systemd-units\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.906593 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906443 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-kubelet-config\") pod \"global-pull-secret-syncer-lkclq\" (UID: \"efd59b32-c96c-4808-9a83-b24cdc2e3dc7\") " pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:03.906593 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906467 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-multus-cni-dir\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.906593 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906497 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-etc-kubernetes\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.906593 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906515 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-systemd-units\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.906593 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906534 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-run-netns\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.906593 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906561 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-host-var-lib-cni-multus\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.906593 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906565 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-ovnkube-config\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.906593 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906591 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-dbus\") pod \"global-pull-secret-syncer-lkclq\" (UID: \"efd59b32-c96c-4808-9a83-b24cdc2e3dc7\") " pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:03.907437 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906599 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-ovnkube-script-lib\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.907437 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906625 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-run-netns\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.907437 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906632 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-kubelet\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.907437 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906658 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96dqc\" (UniqueName: \"kubernetes.io/projected/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-kube-api-access-96dqc\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.907437 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906688 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-multus-socket-dir-parent\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.907437 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906702 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-kubelet\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.907437 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906719 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-multus-conf-dir\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.907437 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906761 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-multus-conf-dir\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.907437 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906769 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-os-release\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.907437 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906796 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-multus-socket-dir-parent\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.907437 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906798 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-run-systemd\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.907437 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906825 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-var-lib-openvswitch\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.907437 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906837 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-os-release\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.907437 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906865 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b57811b-9cd7-40e0-b4c4-3b24a5822001-cni-binary-copy\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.907437 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906885 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-var-lib-openvswitch\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.907437 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906888 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-slash\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.907437 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906913 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-node-log\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.907437 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906925 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-run-systemd\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908001 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906977 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-hostroot\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.908001 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906978 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-slash\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908001 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907009 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-env-overrides\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908001 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907025 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-hostroot\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.908001 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.906978 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-node-log\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908001 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907040 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret\") pod \"global-pull-secret-syncer-lkclq\" (UID: \"efd59b32-c96c-4808-9a83-b24cdc2e3dc7\") " pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:03.908001 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907068 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-run-ovn-kubernetes\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908001 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907101 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908001 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907144 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-system-cni-dir\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.908001 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:03.907156 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 16:17:03.908001 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907169 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-host-run-multus-certs\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.908001 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907191 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908001 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907196 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-etc-openvswitch\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908001 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907156 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-run-ovn-kubernetes\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908001 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:03.907232 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret podName:efd59b32-c96c-4808-9a83-b24cdc2e3dc7 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:04.407210418 +0000 UTC m=+3.079744912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret") pod "global-pull-secret-syncer-lkclq" (UID: "efd59b32-c96c-4808-9a83-b24cdc2e3dc7") : object "kube-system"/"original-pull-secret" not registered Jul 27 16:17:03.908001 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907244 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-etc-openvswitch\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908001 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907255 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-ovnkube-config\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907275 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-ovnkube-script-lib\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907278 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-system-cni-dir\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.908820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907293 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-host-run-multus-certs\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.908820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907315 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-run-openvswitch\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907353 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-host-run-netns\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.908820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907387 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk6vj\" (UniqueName: \"kubernetes.io/projected/8b57811b-9cd7-40e0-b4c4-3b24a5822001-kube-api-access-vk6vj\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.908820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907420 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-run-openvswitch\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907429 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b57811b-9cd7-40e0-b4c4-3b24a5822001-host-run-netns\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.908820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907432 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-run-ovn\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907488 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b57811b-9cd7-40e0-b4c4-3b24a5822001-cni-binary-copy\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.908820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907492 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-cni-bin\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907489 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-run-ovn\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907527 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b57811b-9cd7-40e0-b4c4-3b24a5822001-multus-daemon-config\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.908820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907552 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-cni-netd\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907555 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-cni-bin\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907629 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-host-cni-netd\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.908820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.907948 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b57811b-9cd7-40e0-b4c4-3b24a5822001-multus-daemon-config\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.908820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.908616 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-env-overrides\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.909903 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.909878 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-ovn-node-metrics-cert\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.917668 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.917605 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk6vj\" (UniqueName: \"kubernetes.io/projected/8b57811b-9cd7-40e0-b4c4-3b24a5822001-kube-api-access-vk6vj\") pod \"multus-9lrvm\" (UID: \"8b57811b-9cd7-40e0-b4c4-3b24a5822001\") " pod="openshift-multus/multus-9lrvm" Jul 27 16:17:03.917668 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.917659 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96dqc\" (UniqueName: \"kubernetes.io/projected/dbedd2ca-c568-40f9-a9d2-be0dae1b803c-kube-api-access-96dqc\") pod \"ovnkube-node-f9gll\" (UID: \"dbedd2ca-c568-40f9-a9d2-be0dae1b803c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:03.940551 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.940519 2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 16:17:03.992251 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.992215 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" Jul 27 16:17:03.998791 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:03.998761 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q9n8r" Jul 27 16:17:04.008979 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.008661 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tzvpv" Jul 27 16:17:04.009842 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:04.009802 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca803c9f_3708_4539_be6c_abef843d77d0.slice/crio-1d7ff5cc6a1571b8a345a3f617fe655b02466f8e0810b3ad5f6955664fe76392 WatchSource:0}: Error finding container 1d7ff5cc6a1571b8a345a3f617fe655b02466f8e0810b3ad5f6955664fe76392: Status 404 returned error can't find the container with id 1d7ff5cc6a1571b8a345a3f617fe655b02466f8e0810b3ad5f6955664fe76392 Jul 27 16:17:04.010563 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:04.010527 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26550e07_19b7_4937_9964_58c93feb47a7.slice/crio-0c3156113e3d58a02f7a03c6e20c6fdb836658e1b7b068d2b2216521ffcba3c6 WatchSource:0}: Error finding container 0c3156113e3d58a02f7a03c6e20c6fdb836658e1b7b068d2b2216521ffcba3c6: Status 404 returned error can't find the container with id 0c3156113e3d58a02f7a03c6e20c6fdb836658e1b7b068d2b2216521ffcba3c6 Jul 27 16:17:04.014810 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.014789 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tnb6j" Jul 27 16:17:04.020224 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:04.020199 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5445bc4e_d01a_48c8_8c71_6541ad9a487e.slice/crio-940afea8506120d38dc70909e0415baff1e25923419cfe40268b093cab6ca744 WatchSource:0}: Error finding container 940afea8506120d38dc70909e0415baff1e25923419cfe40268b093cab6ca744: Status 404 returned error can't find the container with id 940afea8506120d38dc70909e0415baff1e25923419cfe40268b093cab6ca744 Jul 27 16:17:04.021827 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.021801 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" Jul 27 16:17:04.026324 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:04.026302 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8bc6ae_9d34_41e4_8079_9149a29e6627.slice/crio-f069a4a7503d5826c27a2747e81bde69de83b2e050d65fc3b95dfff85ec73ba5 WatchSource:0}: Error finding container f069a4a7503d5826c27a2747e81bde69de83b2e050d65fc3b95dfff85ec73ba5: Status 404 returned error can't find the container with id f069a4a7503d5826c27a2747e81bde69de83b2e050d65fc3b95dfff85ec73ba5 Jul 27 16:17:04.031704 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:04.031684 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f1cd14_e645_4eb1_bef3_22089007cf7f.slice/crio-d336491fc7ef0cef95e5b6045f00692d1180a3fb29d815c898a439c61d16968e WatchSource:0}: Error finding container d336491fc7ef0cef95e5b6045f00692d1180a3fb29d815c898a439c61d16968e: Status 404 returned error can't find the container with id d336491fc7ef0cef95e5b6045f00692d1180a3fb29d815c898a439c61d16968e Jul 27 16:17:04.033820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.033801 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9fnfd" Jul 27 16:17:04.042361 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.042185 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dkzxx" Jul 27 16:17:04.042494 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:04.042471 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a27900_e3d3_44c2_a7da_7c3ff58b7cfc.slice/crio-a185f21a2fa9937f295bba4a0f47436ef5efa4f75a815d18cdfd23f5d99b2277 WatchSource:0}: Error finding container a185f21a2fa9937f295bba4a0f47436ef5efa4f75a815d18cdfd23f5d99b2277: Status 404 returned error can't find the container with id a185f21a2fa9937f295bba4a0f47436ef5efa4f75a815d18cdfd23f5d99b2277 Jul 27 16:17:04.050590 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.050571 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9lrvm" Jul 27 16:17:04.051213 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:04.051189 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb765f995_1ea5_4a72_b3ad_c661cd537410.slice/crio-8b2968d0dad9d84f4dbb4106e138752a902cc9070f5ffbf61a33b774515b0699 WatchSource:0}: Error finding container 8b2968d0dad9d84f4dbb4106e138752a902cc9070f5ffbf61a33b774515b0699: Status 404 returned error can't find the container with id 8b2968d0dad9d84f4dbb4106e138752a902cc9070f5ffbf61a33b774515b0699 Jul 27 16:17:04.057512 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.057488 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:04.059741 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:04.059716 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b57811b_9cd7_40e0_b4c4_3b24a5822001.slice/crio-55735b6e8e858594f5f22bd703f6477b28ff096537fac408c5d4e1fbc4f94921 WatchSource:0}: Error finding container 55735b6e8e858594f5f22bd703f6477b28ff096537fac408c5d4e1fbc4f94921: Status 404 returned error can't find the container with id 55735b6e8e858594f5f22bd703f6477b28ff096537fac408c5d4e1fbc4f94921 Jul 27 16:17:04.067672 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:04.067648 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbedd2ca_c568_40f9_a9d2_be0dae1b803c.slice/crio-abd9bc43387f2ef8e205f89ea7e213f4303ff9202fb5bf684334d9829e0b4885 WatchSource:0}: Error finding container abd9bc43387f2ef8e205f89ea7e213f4303ff9202fb5bf684334d9829e0b4885: Status 404 returned error can't find the container with id abd9bc43387f2ef8e205f89ea7e213f4303ff9202fb5bf684334d9829e0b4885 Jul 27 16:17:04.311252 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.311221 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs\") pod \"network-metrics-daemon-mg2dr\" (UID: \"fc3e4156-6cfc-430a-8842-a9d801cfb9f0\") " pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:04.311449 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.311276 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7bxt\" (UniqueName: \"kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt\") pod \"network-check-target-26l7z\" (UID: \"b1c1e78a-1087-4293-aca2-a34e14ee090b\") " pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:04.311449 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:04.311393 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:17:04.311449 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:04.311407 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 16:17:04.311449 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:04.311424 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 16:17:04.311449 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:04.311437 2550 projected.go:196] Error preparing data for projected volume kube-api-access-s7bxt for pod openshift-network-diagnostics/network-check-target-26l7z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:17:04.311662 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:04.311466 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs podName:fc3e4156-6cfc-430a-8842-a9d801cfb9f0 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:05.311447092 +0000 UTC m=+3.983981587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs") pod "network-metrics-daemon-mg2dr" (UID: "fc3e4156-6cfc-430a-8842-a9d801cfb9f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:17:04.311662 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:04.311505 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt podName:b1c1e78a-1087-4293-aca2-a34e14ee090b nodeName:}" failed. No retries permitted until 2026-07-27 16:17:05.31149741 +0000 UTC m=+3.984031889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s7bxt" (UniqueName: "kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt") pod "network-check-target-26l7z" (UID: "b1c1e78a-1087-4293-aca2-a34e14ee090b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:17:04.412694 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.412646 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret\") pod \"global-pull-secret-syncer-lkclq\" (UID: \"efd59b32-c96c-4808-9a83-b24cdc2e3dc7\") " pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:04.412870 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:04.412852 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 16:17:04.412971 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:04.412957 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret podName:efd59b32-c96c-4808-9a83-b24cdc2e3dc7 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:05.412933273 +0000 UTC m=+4.085467755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret") pod "global-pull-secret-syncer-lkclq" (UID: "efd59b32-c96c-4808-9a83-b24cdc2e3dc7") : object "kube-system"/"original-pull-secret" not registered Jul 27 16:17:04.752595 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.752443 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 16:12:04 +0000 UTC" deadline="2027-12-26 02:13:00.660106502 +0000 UTC" Jul 27 16:17:04.752595 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.752491 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12393h55m55.907624382s" Jul 27 16:17:04.857604 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.857563 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9lrvm" event={"ID":"8b57811b-9cd7-40e0-b4c4-3b24a5822001","Type":"ContainerStarted","Data":"55735b6e8e858594f5f22bd703f6477b28ff096537fac408c5d4e1fbc4f94921"} Jul 27 16:17:04.861902 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.861867 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dkzxx" event={"ID":"b765f995-1ea5-4a72-b3ad-c661cd537410","Type":"ContainerStarted","Data":"8b2968d0dad9d84f4dbb4106e138752a902cc9070f5ffbf61a33b774515b0699"} Jul 27 16:17:04.864823 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.864786 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" event={"ID":"f1f1cd14-e645-4eb1-bef3-22089007cf7f","Type":"ContainerStarted","Data":"d336491fc7ef0cef95e5b6045f00692d1180a3fb29d815c898a439c61d16968e"} Jul 27 16:17:04.866884 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.866858 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tnb6j" event={"ID":"9b8bc6ae-9d34-41e4-8079-9149a29e6627","Type":"ContainerStarted","Data":"f069a4a7503d5826c27a2747e81bde69de83b2e050d65fc3b95dfff85ec73ba5"} Jul 27 16:17:04.869046 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.868984 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9n8r" event={"ID":"26550e07-19b7-4937-9964-58c93feb47a7","Type":"ContainerStarted","Data":"0c3156113e3d58a02f7a03c6e20c6fdb836658e1b7b068d2b2216521ffcba3c6"} Jul 27 16:17:04.872116 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.872080 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" event={"ID":"dbedd2ca-c568-40f9-a9d2-be0dae1b803c","Type":"ContainerStarted","Data":"abd9bc43387f2ef8e205f89ea7e213f4303ff9202fb5bf684334d9829e0b4885"} Jul 27 16:17:04.875263 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.875234 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9fnfd" event={"ID":"92a27900-e3d3-44c2-a7da-7c3ff58b7cfc","Type":"ContainerStarted","Data":"a185f21a2fa9937f295bba4a0f47436ef5efa4f75a815d18cdfd23f5d99b2277"} Jul 27 16:17:04.877420 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.877394 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tzvpv" event={"ID":"5445bc4e-d01a-48c8-8c71-6541ad9a487e","Type":"ContainerStarted","Data":"940afea8506120d38dc70909e0415baff1e25923419cfe40268b093cab6ca744"} Jul 27 16:17:04.879615 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:04.879581 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" event={"ID":"ca803c9f-3708-4539-be6c-abef843d77d0","Type":"ContainerStarted","Data":"1d7ff5cc6a1571b8a345a3f617fe655b02466f8e0810b3ad5f6955664fe76392"} Jul 27 16:17:05.029845 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:05.029616 2550 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 16:17:05.320139 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:05.320098 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs\") pod \"network-metrics-daemon-mg2dr\" (UID: \"fc3e4156-6cfc-430a-8842-a9d801cfb9f0\") " pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:05.320336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:05.320179 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7bxt\" (UniqueName: \"kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt\") pod \"network-check-target-26l7z\" (UID: \"b1c1e78a-1087-4293-aca2-a34e14ee090b\") " pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:05.320405 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:05.320347 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 16:17:05.320405 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:05.320366 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 16:17:05.320405 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:05.320381 2550 projected.go:196] Error preparing data for projected volume kube-api-access-s7bxt for pod openshift-network-diagnostics/network-check-target-26l7z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:17:05.320566 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:05.320446 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt podName:b1c1e78a-1087-4293-aca2-a34e14ee090b nodeName:}" failed. No retries permitted until 2026-07-27 16:17:07.320425942 +0000 UTC m=+5.992960430 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s7bxt" (UniqueName: "kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt") pod "network-check-target-26l7z" (UID: "b1c1e78a-1087-4293-aca2-a34e14ee090b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:17:05.320566 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:05.320525 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:17:05.320566 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:05.320563 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs podName:fc3e4156-6cfc-430a-8842-a9d801cfb9f0 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:07.320548865 +0000 UTC m=+5.993083343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs") pod "network-metrics-daemon-mg2dr" (UID: "fc3e4156-6cfc-430a-8842-a9d801cfb9f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:17:05.420806 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:05.420769 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret\") pod \"global-pull-secret-syncer-lkclq\" (UID: \"efd59b32-c96c-4808-9a83-b24cdc2e3dc7\") " pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:05.421020 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:05.420915 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 16:17:05.421020 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:05.420999 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret podName:efd59b32-c96c-4808-9a83-b24cdc2e3dc7 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:07.420981185 +0000 UTC m=+6.093515682 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret") pod "global-pull-secret-syncer-lkclq" (UID: "efd59b32-c96c-4808-9a83-b24cdc2e3dc7") : object "kube-system"/"original-pull-secret" not registered Jul 27 16:17:05.753465 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:05.753358 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 16:12:04 +0000 UTC" deadline="2028-01-01 05:55:59.967927991 +0000 UTC" Jul 27 16:17:05.753465 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:05.753424 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12541h38m54.214508548s" Jul 27 16:17:05.847045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:05.844494 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:05.847045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:05.844494 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:05.847045 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:05.844664 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:05.847045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:05.844684 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:05.847045 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:05.844794 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:05.847045 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:05.844891 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:05.890203 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:05.890142 2550 generic.go:356] "Generic (PLEG): container finished" podID="158785e5c4cf0d4aef5b4d9ab831c0ca" containerID="b9bc4cc51937afb94d82cc36af4ea02f6230fcd5de9319f28377113100f9bfb6" exitCode=0 Jul 27 16:17:05.890203 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:05.890200 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal" event={"ID":"158785e5c4cf0d4aef5b4d9ab831c0ca","Type":"ContainerDied","Data":"b9bc4cc51937afb94d82cc36af4ea02f6230fcd5de9319f28377113100f9bfb6"} Jul 27 16:17:07.337979 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:07.337939 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7bxt\" (UniqueName: \"kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt\") pod \"network-check-target-26l7z\" (UID: \"b1c1e78a-1087-4293-aca2-a34e14ee090b\") " pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:07.338516 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:07.338020 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs\") pod \"network-metrics-daemon-mg2dr\" (UID: \"fc3e4156-6cfc-430a-8842-a9d801cfb9f0\") " pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:07.338516 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:07.338112 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 16:17:07.338516 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:07.338133 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 16:17:07.338516 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:07.338145 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:17:07.338516 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:07.338147 2550 projected.go:196] Error preparing data for projected volume kube-api-access-s7bxt for pod openshift-network-diagnostics/network-check-target-26l7z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:17:07.338516 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:07.338199 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs podName:fc3e4156-6cfc-430a-8842-a9d801cfb9f0 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:11.338182131 +0000 UTC m=+10.010716626 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs") pod "network-metrics-daemon-mg2dr" (UID: "fc3e4156-6cfc-430a-8842-a9d801cfb9f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:17:07.338516 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:07.338217 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt podName:b1c1e78a-1087-4293-aca2-a34e14ee090b nodeName:}" failed. No retries permitted until 2026-07-27 16:17:11.338207776 +0000 UTC m=+10.010742261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s7bxt" (UniqueName: "kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt") pod "network-check-target-26l7z" (UID: "b1c1e78a-1087-4293-aca2-a34e14ee090b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:17:07.439450 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:07.439412 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret\") pod \"global-pull-secret-syncer-lkclq\" (UID: \"efd59b32-c96c-4808-9a83-b24cdc2e3dc7\") " pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:07.439633 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:07.439612 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 16:17:07.439696 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:07.439674 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret podName:efd59b32-c96c-4808-9a83-b24cdc2e3dc7 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:11.439655477 +0000 UTC m=+10.112189965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret") pod "global-pull-secret-syncer-lkclq" (UID: "efd59b32-c96c-4808-9a83-b24cdc2e3dc7") : object "kube-system"/"original-pull-secret" not registered Jul 27 16:17:07.844057 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:07.843597 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:07.844057 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:07.843712 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:07.844057 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:07.844004 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:07.844379 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:07.844120 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:07.844379 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:07.844166 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:07.844379 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:07.844242 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:08.560193 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:08.560169 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefedf659439c0f4b07ff2be7c9f4e1bd.slice/crio-279b5846e6a2fd5f5c4fdc6b9deaa96835a0535807d67f19041acf9f051d0ba2 WatchSource:0}: Error finding container 279b5846e6a2fd5f5c4fdc6b9deaa96835a0535807d67f19041acf9f051d0ba2: Status 404 returned error can't find the container with id 279b5846e6a2fd5f5c4fdc6b9deaa96835a0535807d67f19041acf9f051d0ba2 Jul 27 16:17:08.899630 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:08.899281 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tzvpv" event={"ID":"5445bc4e-d01a-48c8-8c71-6541ad9a487e","Type":"ContainerStarted","Data":"f60c84d1bc66e0c2ccdd37fda0e1aab850c68c13f62e6ecbe211d9f454c48a7f"} Jul 27 16:17:08.902753 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:08.902708 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" event={"ID":"f1f1cd14-e645-4eb1-bef3-22089007cf7f","Type":"ContainerStarted","Data":"bd8da49a987311ab441d8b17f5d063d8d354c21bec54a42dfca8c21927d9aeff"} Jul 27 16:17:08.906470 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:08.906423 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tnb6j" event={"ID":"9b8bc6ae-9d34-41e4-8079-9149a29e6627","Type":"ContainerStarted","Data":"edf0697cdf35277fb5e56a8cc895ee08e6e2dd58807d447d188fad2b53a69e2a"} Jul 27 16:17:08.911171 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:08.911143 2550 generic.go:356] "Generic (PLEG): container finished" podID="26550e07-19b7-4937-9964-58c93feb47a7" containerID="ae50da855991a14d0f940aa91892f205a28d7d44658f7849a8edc8d0208dbfd9" exitCode=0 Jul 27 16:17:08.911267 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:08.911224 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9n8r" event={"ID":"26550e07-19b7-4937-9964-58c93feb47a7","Type":"ContainerDied","Data":"ae50da855991a14d0f940aa91892f205a28d7d44658f7849a8edc8d0208dbfd9"} Jul 27 16:17:08.917193 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:08.917164 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.ec2.internal" event={"ID":"efedf659439c0f4b07ff2be7c9f4e1bd","Type":"ContainerStarted","Data":"279b5846e6a2fd5f5c4fdc6b9deaa96835a0535807d67f19041acf9f051d0ba2"} Jul 27 16:17:08.921378 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:08.920879 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tzvpv" podStartSLOduration=2.410686177 podStartE2EDuration="6.920861203s" totalImagesPullingTime="4.510175026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:08.920222774 +0000 UTC m=+7.592757276" watchObservedRunningTime="2026-07-27 16:17:08.920861203 +0000 UTC m=+7.593395708" Jul 27 16:17:08.923876 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:08.923399 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal" event={"ID":"158785e5c4cf0d4aef5b4d9ab831c0ca","Type":"ContainerStarted","Data":"e0bae02921c3f0dddb34932bd9ff4c7ab4287fbc1d9ec5b6facfebfdf788a614"} Jul 27 16:17:09.024114 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:09.023970 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tnb6j" podStartSLOduration=2.514225802 podStartE2EDuration="7.023945775s" totalImagesPullingTime="4.509719973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:08.973486179 +0000 UTC m=+7.646020685" watchObservedRunningTime="2026-07-27 16:17:09.023945775 +0000 UTC m=+7.696480279" Jul 27 16:17:09.487887 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:09.487795 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tzvpv" Jul 27 16:17:09.488643 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:09.488614 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tzvpv" Jul 27 16:17:09.513666 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:09.513607 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.ec2.internal" podStartSLOduration=6.513585248 podStartE2EDuration="6.513585248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:09.024151537 +0000 UTC m=+7.696686040" watchObservedRunningTime="2026-07-27 16:17:09.513585248 +0000 UTC m=+8.186119752" Jul 27 16:17:09.844625 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:09.843792 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:09.844625 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:09.843968 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:09.844625 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:09.844466 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:09.844625 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:09.844568 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:09.845525 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:09.845388 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:09.845525 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:09.845488 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:10.930071 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:10.929526 2550 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 27 16:17:11.377745 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:11.376719 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs\") pod \"network-metrics-daemon-mg2dr\" (UID: \"fc3e4156-6cfc-430a-8842-a9d801cfb9f0\") " pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:11.377745 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:11.376792 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7bxt\" (UniqueName: \"kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt\") pod \"network-check-target-26l7z\" (UID: \"b1c1e78a-1087-4293-aca2-a34e14ee090b\") " pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:11.377745 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:11.376969 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 16:17:11.377745 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:11.376991 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 16:17:11.377745 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:11.377004 2550 projected.go:196] Error preparing data for projected volume kube-api-access-s7bxt for pod openshift-network-diagnostics/network-check-target-26l7z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:17:11.377745 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:11.377064 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt podName:b1c1e78a-1087-4293-aca2-a34e14ee090b nodeName:}" failed. No retries permitted until 2026-07-27 16:17:19.377043484 +0000 UTC m=+18.049577978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s7bxt" (UniqueName: "kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt") pod "network-check-target-26l7z" (UID: "b1c1e78a-1087-4293-aca2-a34e14ee090b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:17:11.377745 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:11.377613 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:17:11.377745 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:11.377668 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs podName:fc3e4156-6cfc-430a-8842-a9d801cfb9f0 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:19.377651437 +0000 UTC m=+18.050185933 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs") pod "network-metrics-daemon-mg2dr" (UID: "fc3e4156-6cfc-430a-8842-a9d801cfb9f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:17:11.477242 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:11.477203 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret\") pod \"global-pull-secret-syncer-lkclq\" (UID: \"efd59b32-c96c-4808-9a83-b24cdc2e3dc7\") " pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:11.477451 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:11.477379 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 16:17:11.477451 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:11.477447 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret podName:efd59b32-c96c-4808-9a83-b24cdc2e3dc7 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:19.477427704 +0000 UTC m=+18.149962204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret") pod "global-pull-secret-syncer-lkclq" (UID: "efd59b32-c96c-4808-9a83-b24cdc2e3dc7") : object "kube-system"/"original-pull-secret" not registered Jul 27 16:17:11.845146 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:11.845108 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:11.845335 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:11.845160 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:11.845335 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:11.845276 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:11.845335 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:11.845300 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:11.845514 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:11.845371 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:11.845514 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:11.845441 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:13.843567 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:13.843528 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:13.844117 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:13.843586 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:13.844117 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:13.843528 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:13.844117 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:13.843697 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:13.844117 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:13.843763 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:13.844117 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:13.843837 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:15.844049 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:15.844015 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:15.844049 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:15.844030 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:15.844563 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:15.844135 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:15.844563 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:15.844208 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:15.844563 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:15.844270 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:15.844563 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:15.844342 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:16.821895 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:16.821858 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tzvpv" Jul 27 16:17:16.822096 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:16.822029 2550 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 27 16:17:16.822476 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:16.822459 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tzvpv" Jul 27 16:17:17.845730 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:17.845697 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:17.846224 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:17.845824 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:17.846224 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:17.845997 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:17.846224 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:17.846006 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:17.846224 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:17.846094 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:17.846224 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:17.846182 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:19.445968 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:19.445936 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs\") pod \"network-metrics-daemon-mg2dr\" (UID: \"fc3e4156-6cfc-430a-8842-a9d801cfb9f0\") " pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:19.446386 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:19.445998 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7bxt\" (UniqueName: \"kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt\") pod \"network-check-target-26l7z\" (UID: \"b1c1e78a-1087-4293-aca2-a34e14ee090b\") " pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:19.446386 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:19.446120 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 16:17:19.446386 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:19.446118 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:17:19.446386 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:19.446135 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 16:17:19.446386 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:19.446154 2550 projected.go:196] Error preparing data for projected volume kube-api-access-s7bxt for pod openshift-network-diagnostics/network-check-target-26l7z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:17:19.446386 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:19.446217 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs podName:fc3e4156-6cfc-430a-8842-a9d801cfb9f0 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:35.446183162 +0000 UTC m=+34.118717643 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs") pod "network-metrics-daemon-mg2dr" (UID: "fc3e4156-6cfc-430a-8842-a9d801cfb9f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:17:19.446386 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:19.446241 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt podName:b1c1e78a-1087-4293-aca2-a34e14ee090b nodeName:}" failed. No retries permitted until 2026-07-27 16:17:35.446228575 +0000 UTC m=+34.118763053 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s7bxt" (UniqueName: "kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt") pod "network-check-target-26l7z" (UID: "b1c1e78a-1087-4293-aca2-a34e14ee090b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:17:19.546850 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:19.546816 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret\") pod \"global-pull-secret-syncer-lkclq\" (UID: \"efd59b32-c96c-4808-9a83-b24cdc2e3dc7\") " pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:19.547042 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:19.546958 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 16:17:19.547042 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:19.547026 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret podName:efd59b32-c96c-4808-9a83-b24cdc2e3dc7 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:35.547011546 +0000 UTC m=+34.219546025 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret") pod "global-pull-secret-syncer-lkclq" (UID: "efd59b32-c96c-4808-9a83-b24cdc2e3dc7") : object "kube-system"/"original-pull-secret" not registered Jul 27 16:17:19.844122 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:19.844091 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:19.844122 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:19.844104 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:19.844365 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:19.844220 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:19.844365 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:19.844347 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:19.844464 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:19.844412 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:19.844515 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:19.844468 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:21.845274 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:21.845021 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:21.845675 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:21.845103 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:21.845675 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:21.845377 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:21.845675 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:21.845129 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:21.845675 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:21.845481 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:21.845675 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:21.845515 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:23.844277 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:23.844242 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:23.844802 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:23.844243 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:23.844802 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:23.844375 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:23.844802 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:23.844453 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:23.844802 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:23.844241 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:23.844802 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:23.844539 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:25.844114 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:25.844081 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:25.844571 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:25.844086 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:25.844571 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:25.844215 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:25.844571 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:25.844311 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:25.844571 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:25.844090 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:25.844571 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:25.844401 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:27.546352 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.546330 2550 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 27 16:17:27.764018 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.763928 2550 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T16:17:27.546347946Z","UUID":"a89d0053-0695-4d69-9e32-1b624410f4cf","Handler":null,"Name":"","Endpoint":""} Jul 27 16:17:27.765326 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.765306 2550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 27 16:17:27.765326 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.765342 2550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 27 16:17:27.845785 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.843484 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:27.845785 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.843506 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:27.845785 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.843564 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:27.845785 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:27.843773 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:27.845785 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:27.843886 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:27.845785 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:27.844011 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:27.964890 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.964859 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" event={"ID":"ca803c9f-3708-4539-be6c-abef843d77d0","Type":"ContainerStarted","Data":"a9545a5487299691bd56ffe894ab8f0e8473ced887000e0264afae5198a26796"} Jul 27 16:17:27.966325 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.966304 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9lrvm" event={"ID":"8b57811b-9cd7-40e0-b4c4-3b24a5822001","Type":"ContainerStarted","Data":"db2aa465cbd7fda1d4152778f4c32ba9a8e18b3cac8b1bf63f97b4ce11723e84"} Jul 27 16:17:27.968125 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.968101 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" event={"ID":"f1f1cd14-e645-4eb1-bef3-22089007cf7f","Type":"ContainerStarted","Data":"5ed46f483f19c06d0fd07ab5b0f133bcd6efc84fbb5f94db07f17ccca8637d19"} Jul 27 16:17:27.970827 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.970802 2550 generic.go:356] "Generic (PLEG): container finished" podID="26550e07-19b7-4937-9964-58c93feb47a7" containerID="4eba381734112bb534d0a5d03da8fa891dc0d301e0c592aa41a73d5d1fdf276a" exitCode=0 Jul 27 16:17:27.970944 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.970880 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9n8r" event={"ID":"26550e07-19b7-4937-9964-58c93feb47a7","Type":"ContainerDied","Data":"4eba381734112bb534d0a5d03da8fa891dc0d301e0c592aa41a73d5d1fdf276a"} Jul 27 16:17:27.972708 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.972685 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.ec2.internal" event={"ID":"efedf659439c0f4b07ff2be7c9f4e1bd","Type":"ContainerStarted","Data":"839f6c752a4942fd7bfd65f14759d3394753b6f2e068e8521b9b075e4ae4a234"} Jul 27 16:17:27.975515 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.975500 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:17:27.975845 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.975824 2550 generic.go:356] "Generic (PLEG): container finished" podID="dbedd2ca-c568-40f9-a9d2-be0dae1b803c" containerID="8cac9f8379525db03667995a80295d5a97065132164cd8999255169b7661008b" exitCode=1 Jul 27 16:17:27.975926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.975887 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" event={"ID":"dbedd2ca-c568-40f9-a9d2-be0dae1b803c","Type":"ContainerStarted","Data":"60b3a1ff3f837b018b8307bbaaddc001a59cd08dd79611958e26516bf7a10f37"} Jul 27 16:17:27.975974 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.975954 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" event={"ID":"dbedd2ca-c568-40f9-a9d2-be0dae1b803c","Type":"ContainerDied","Data":"8cac9f8379525db03667995a80295d5a97065132164cd8999255169b7661008b"} Jul 27 16:17:27.976014 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.976000 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" event={"ID":"dbedd2ca-c568-40f9-a9d2-be0dae1b803c","Type":"ContainerStarted","Data":"fabbf621cbfd78f13ab876be096bbddbbbf499c144abd4a64a53a3174b65ba46"} Jul 27 16:17:27.977479 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.977453 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9fnfd" event={"ID":"92a27900-e3d3-44c2-a7da-7c3ff58b7cfc","Type":"ContainerStarted","Data":"e302b0b271dca4c48bc9c257a889ff54ae6864eb4c9ca65fbc90ab56c9faccb3"} Jul 27 16:17:27.985742 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:27.985705 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4mx8h" podStartSLOduration=2.603934196 podStartE2EDuration="25.985692941s" totalImagesPullingTime="23.381758745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:27.985638138 +0000 UTC m=+26.658172639" watchObservedRunningTime="2026-07-27 16:17:27.985692941 +0000 UTC m=+26.658227439" Jul 27 16:17:28.032317 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:28.032282 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9fnfd" podStartSLOduration=2.70537029 podStartE2EDuration="26.032266329s" totalImagesPullingTime="23.326896039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:28.031639658 +0000 UTC m=+26.704174158" watchObservedRunningTime="2026-07-27 16:17:28.032266329 +0000 UTC m=+26.704800827" Jul 27 16:17:28.047649 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:28.047615 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.ec2.internal" podStartSLOduration=25.047600388 podStartE2EDuration="25.047600388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:28.047451469 +0000 UTC m=+26.719985969" watchObservedRunningTime="2026-07-27 16:17:28.047600388 +0000 UTC m=+26.720134887" Jul 27 16:17:28.067046 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:28.066993 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9lrvm" podStartSLOduration=2.699662501 podStartE2EDuration="26.066974065s" totalImagesPullingTime="23.367311564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:28.065934663 +0000 UTC m=+26.738469163" watchObservedRunningTime="2026-07-27 16:17:28.066974065 +0000 UTC m=+26.739508565" Jul 27 16:17:28.980641 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:28.980547 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dkzxx" event={"ID":"b765f995-1ea5-4a72-b3ad-c661cd537410","Type":"ContainerStarted","Data":"aeec9530eab5fd324a108e2f0406178cf2ff95e3149379f21ef189a3cf7ebe4f"} Jul 27 16:17:28.982649 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:28.982625 2550 generic.go:356] "Generic (PLEG): container finished" podID="26550e07-19b7-4937-9964-58c93feb47a7" containerID="8866ada780c4ee45818323deced40d3778b520c27a5493eedf2719614379404c" exitCode=0 Jul 27 16:17:28.982765 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:28.982704 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9n8r" event={"ID":"26550e07-19b7-4937-9964-58c93feb47a7","Type":"ContainerDied","Data":"8866ada780c4ee45818323deced40d3778b520c27a5493eedf2719614379404c"} Jul 27 16:17:28.985474 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:28.985373 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:17:28.985825 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:28.985801 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" event={"ID":"dbedd2ca-c568-40f9-a9d2-be0dae1b803c","Type":"ContainerStarted","Data":"50f15f1498e828fd03ee0fb25a79810ada25e086472d6c688bb991c330ab6b7d"} Jul 27 16:17:28.985945 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:28.985859 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" event={"ID":"dbedd2ca-c568-40f9-a9d2-be0dae1b803c","Type":"ContainerStarted","Data":"cb7965f13f5010e9fb4252984cd33f86a7024d8423ea3b0e5add77f23a7071e1"} Jul 27 16:17:28.985945 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:28.985889 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" event={"ID":"dbedd2ca-c568-40f9-a9d2-be0dae1b803c","Type":"ContainerStarted","Data":"75200f128e3a9249e08796087e987b99df34d32b549412ab29b3849b973a270b"} Jul 27 16:17:28.999702 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:28.999665 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dkzxx" podStartSLOduration=3.681789213 podStartE2EDuration="26.999652256s" totalImagesPullingTime="23.317863043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:28.999336832 +0000 UTC m=+27.671871333" watchObservedRunningTime="2026-07-27 16:17:28.999652256 +0000 UTC m=+27.672186753" Jul 27 16:17:29.847064 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:29.846886 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:29.847210 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:29.846887 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:29.847210 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:29.847136 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:29.847292 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:29.847217 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:29.847292 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:29.846936 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:29.847357 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:29.847292 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:29.990678 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:29.990592 2550 generic.go:356] "Generic (PLEG): container finished" podID="26550e07-19b7-4937-9964-58c93feb47a7" containerID="d1668b0bdafc8f86e377d709bdb9462ee9bc235fad177e35349b78e60a20fbee" exitCode=0 Jul 27 16:17:29.991061 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:29.990674 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9n8r" event={"ID":"26550e07-19b7-4937-9964-58c93feb47a7","Type":"ContainerDied","Data":"d1668b0bdafc8f86e377d709bdb9462ee9bc235fad177e35349b78e60a20fbee"} Jul 27 16:17:30.996965 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:30.996928 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:17:30.997555 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:30.997398 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" event={"ID":"dbedd2ca-c568-40f9-a9d2-be0dae1b803c","Type":"ContainerStarted","Data":"78c2ea1adb6223dabeae1434da9ab5f3138afb5b6c877d5d1278a7d0dd102fe6"} Jul 27 16:17:31.844543 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:31.844514 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:31.844653 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:31.844612 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:31.844653 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:31.844623 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:31.844740 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:31.844693 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:31.844782 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:31.844709 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:31.844853 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:31.844834 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:32.002525 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:32.002488 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" event={"ID":"f1f1cd14-e645-4eb1-bef3-22089007cf7f","Type":"ContainerStarted","Data":"36a4e1deba1b810b1b116b72fadaf06b43a3c816f0d9883a32c11ea0a62d529a"} Jul 27 16:17:32.025448 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:32.025363 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h69bx" podStartSLOduration=7.02302975 podStartE2EDuration="30.025345687s" totalImagesPullingTime="23.002315937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:32.024848505 +0000 UTC m=+30.697383004" watchObservedRunningTime="2026-07-27 16:17:32.025345687 +0000 UTC m=+30.697880183" Jul 27 16:17:33.846965 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:33.846929 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:33.846965 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:33.846954 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:33.846965 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:33.846929 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:33.847510 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:33.847068 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:33.847510 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:33.847149 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:33.847510 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:33.847235 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:35.013369 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:35.013200 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:17:35.013848 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:35.013752 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" event={"ID":"dbedd2ca-c568-40f9-a9d2-be0dae1b803c","Type":"ContainerStarted","Data":"05656048bbc180b151c887326e09fa799833ac7bd67476e6441eda8126d47c6a"} Jul 27 16:17:35.014005 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:35.013977 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:35.014169 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:35.014152 2550 scope.go:117] "RemoveContainer" containerID="8cac9f8379525db03667995a80295d5a97065132164cd8999255169b7661008b" Jul 27 16:17:35.016756 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:35.016735 2550 generic.go:356] "Generic (PLEG): container finished" podID="26550e07-19b7-4937-9964-58c93feb47a7" containerID="319ca88c0940655bad0432a88d85364190651e3c44358a75f32d5ce19713819c" exitCode=0 Jul 27 16:17:35.016860 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:35.016767 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9n8r" event={"ID":"26550e07-19b7-4937-9964-58c93feb47a7","Type":"ContainerDied","Data":"319ca88c0940655bad0432a88d85364190651e3c44358a75f32d5ce19713819c"} Jul 27 16:17:35.030186 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:35.030169 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:35.459511 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:35.459425 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7bxt\" (UniqueName: \"kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt\") pod \"network-check-target-26l7z\" (UID: \"b1c1e78a-1087-4293-aca2-a34e14ee090b\") " pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:35.459511 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:35.459480 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs\") pod \"network-metrics-daemon-mg2dr\" (UID: \"fc3e4156-6cfc-430a-8842-a9d801cfb9f0\") " pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:35.459935 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:35.459586 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:17:35.459935 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:35.459611 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 16:17:35.459935 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:35.459640 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 16:17:35.459935 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:35.459644 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs podName:fc3e4156-6cfc-430a-8842-a9d801cfb9f0 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:07.459627776 +0000 UTC m=+66.132162257 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs") pod "network-metrics-daemon-mg2dr" (UID: "fc3e4156-6cfc-430a-8842-a9d801cfb9f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:17:35.459935 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:35.459653 2550 projected.go:196] Error preparing data for projected volume kube-api-access-s7bxt for pod openshift-network-diagnostics/network-check-target-26l7z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:17:35.459935 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:35.459708 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt podName:b1c1e78a-1087-4293-aca2-a34e14ee090b nodeName:}" failed. No retries permitted until 2026-07-27 16:18:07.459687418 +0000 UTC m=+66.132221894 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s7bxt" (UniqueName: "kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt") pod "network-check-target-26l7z" (UID: "b1c1e78a-1087-4293-aca2-a34e14ee090b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:17:35.560434 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:35.560399 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret\") pod \"global-pull-secret-syncer-lkclq\" (UID: \"efd59b32-c96c-4808-9a83-b24cdc2e3dc7\") " pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:35.560602 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:35.560555 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 16:17:35.560646 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:35.560634 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret podName:efd59b32-c96c-4808-9a83-b24cdc2e3dc7 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:07.560618203 +0000 UTC m=+66.233152680 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret") pod "global-pull-secret-syncer-lkclq" (UID: "efd59b32-c96c-4808-9a83-b24cdc2e3dc7") : object "kube-system"/"original-pull-secret" not registered Jul 27 16:17:35.843480 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:35.843446 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:35.843683 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:35.843446 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:35.843683 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:35.843551 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:35.843683 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:35.843456 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:35.843851 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:35.843623 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:35.843851 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:35.843742 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:35.968468 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:35.968439 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mg2dr"] Jul 27 16:17:35.969842 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:35.969814 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-26l7z"] Jul 27 16:17:35.970481 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:35.970461 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lkclq"] Jul 27 16:17:36.022200 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:36.022167 2550 generic.go:356] "Generic (PLEG): container finished" podID="26550e07-19b7-4937-9964-58c93feb47a7" containerID="92e56e34c71bc4fe5092403baf034ea105c2d25d8e966663f0b6f26139d3631b" exitCode=0 Jul 27 16:17:36.022627 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:36.022253 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9n8r" event={"ID":"26550e07-19b7-4937-9964-58c93feb47a7","Type":"ContainerDied","Data":"92e56e34c71bc4fe5092403baf034ea105c2d25d8e966663f0b6f26139d3631b"} Jul 27 16:17:36.025724 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:36.025693 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:17:36.026126 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:36.026110 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:36.026223 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:36.026126 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:36.026223 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:36.026140 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:36.026223 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:36.026146 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" event={"ID":"dbedd2ca-c568-40f9-a9d2-be0dae1b803c","Type":"ContainerStarted","Data":"4d250ef3fc84c111638e5be68afe116fd57f541c1d69503a8e84ec851ac63950"} Jul 27 16:17:36.026223 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:36.026212 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:36.026467 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:36.026450 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:36.026586 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:36.026545 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:36.026640 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:36.026588 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:36.026697 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:36.026660 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:36.042322 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:36.042171 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:17:36.079161 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:36.079120 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" podStartSLOduration=9.662196178 podStartE2EDuration="33.07910406s" totalImagesPullingTime="23.416907882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:36.077476506 +0000 UTC m=+34.750011029" watchObservedRunningTime="2026-07-27 16:17:36.07910406 +0000 UTC m=+34.751638558" Jul 27 16:17:37.031953 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:37.031893 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9n8r" event={"ID":"26550e07-19b7-4937-9964-58c93feb47a7","Type":"ContainerStarted","Data":"be3f80921a7e171f258e539a3c65626d8600d51747f222c1ab727139c20d8880"} Jul 27 16:17:37.056201 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:37.056158 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q9n8r" podStartSLOduration=6.401691945 podStartE2EDuration="35.056144306s" totalImagesPullingTime="28.654452361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:37.055796714 +0000 UTC m=+35.728331213" watchObservedRunningTime="2026-07-27 16:17:37.056144306 +0000 UTC m=+35.728678805" Jul 27 16:17:37.843825 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:37.843794 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:37.844030 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:37.843845 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:37.844030 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:37.843845 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:37.844030 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:37.843932 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:37.844030 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:37.843971 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:37.844179 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:37.844030 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:39.850021 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:39.849979 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:39.850543 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:39.849978 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:39.850543 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:39.850102 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg2dr" podUID="fc3e4156-6cfc-430a-8842-a9d801cfb9f0" Jul 27 16:17:39.850543 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:39.850185 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:39.850543 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:39.850184 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26l7z" podUID="b1c1e78a-1087-4293-aca2-a34e14ee090b" Jul 27 16:17:39.850543 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:39.850265 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkclq" podUID="efd59b32-c96c-4808-9a83-b24cdc2e3dc7" Jul 27 16:17:41.172745 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.172714 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-70.ec2.internal" event="NodeReady" Jul 27 16:17:41.173223 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.172865 2550 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 27 16:17:41.217841 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.217760 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-f7sqd"] Jul 27 16:17:41.235836 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.235808 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dfvc2"] Jul 27 16:17:41.235997 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.235950 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f7sqd" Jul 27 16:17:41.238651 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.238630 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 27 16:17:41.238779 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.238632 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z2mnq\"" Jul 27 16:17:41.238779 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.238632 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 27 16:17:41.260068 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.260047 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f7sqd"] Jul 27 16:17:41.260172 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.260081 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dfvc2"] Jul 27 16:17:41.260172 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.260147 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dfvc2" Jul 27 16:17:41.262824 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.262806 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 27 16:17:41.262961 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.262830 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9rtzs\"" Jul 27 16:17:41.262961 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.262830 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 27 16:17:41.262961 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.262879 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 27 16:17:41.403664 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.403628 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26b55f2f-e26a-4918-83b8-d49688d388fd-cert\") pod \"ingress-canary-dfvc2\" (UID: \"26b55f2f-e26a-4918-83b8-d49688d388fd\") " pod="openshift-ingress-canary/ingress-canary-dfvc2" Jul 27 16:17:41.403664 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.403662 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jfzl\" (UniqueName: \"kubernetes.io/projected/26b55f2f-e26a-4918-83b8-d49688d388fd-kube-api-access-7jfzl\") pod \"ingress-canary-dfvc2\" (UID: \"26b55f2f-e26a-4918-83b8-d49688d388fd\") " pod="openshift-ingress-canary/ingress-canary-dfvc2" Jul 27 16:17:41.403869 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.403701 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls\") pod \"dns-default-f7sqd\" (UID: \"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac\") " pod="openshift-dns/dns-default-f7sqd" Jul 27 16:17:41.403869 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.403719 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-tmp-dir\") pod \"dns-default-f7sqd\" (UID: \"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac\") " pod="openshift-dns/dns-default-f7sqd" Jul 27 16:17:41.403869 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.403758 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9xn5\" (UniqueName: \"kubernetes.io/projected/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-kube-api-access-r9xn5\") pod \"dns-default-f7sqd\" (UID: \"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac\") " pod="openshift-dns/dns-default-f7sqd" Jul 27 16:17:41.403869 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.403778 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-config-volume\") pod \"dns-default-f7sqd\" (UID: \"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac\") " pod="openshift-dns/dns-default-f7sqd" Jul 27 16:17:41.504956 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.504867 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26b55f2f-e26a-4918-83b8-d49688d388fd-cert\") pod \"ingress-canary-dfvc2\" (UID: \"26b55f2f-e26a-4918-83b8-d49688d388fd\") " pod="openshift-ingress-canary/ingress-canary-dfvc2" Jul 27 16:17:41.504956 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.504918 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jfzl\" (UniqueName: \"kubernetes.io/projected/26b55f2f-e26a-4918-83b8-d49688d388fd-kube-api-access-7jfzl\") pod \"ingress-canary-dfvc2\" (UID: \"26b55f2f-e26a-4918-83b8-d49688d388fd\") " pod="openshift-ingress-canary/ingress-canary-dfvc2" Jul 27 16:17:41.505116 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.504966 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls\") pod \"dns-default-f7sqd\" (UID: \"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac\") " pod="openshift-dns/dns-default-f7sqd" Jul 27 16:17:41.505116 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.504985 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-tmp-dir\") pod \"dns-default-f7sqd\" (UID: \"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac\") " pod="openshift-dns/dns-default-f7sqd" Jul 27 16:17:41.505116 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.505000 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9xn5\" (UniqueName: \"kubernetes.io/projected/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-kube-api-access-r9xn5\") pod \"dns-default-f7sqd\" (UID: \"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac\") " pod="openshift-dns/dns-default-f7sqd" Jul 27 16:17:41.505116 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.505046 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-config-volume\") pod \"dns-default-f7sqd\" (UID: \"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac\") " pod="openshift-dns/dns-default-f7sqd" Jul 27 16:17:41.505295 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:41.505216 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 16:17:41.505295 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:41.505290 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls podName:76c82e0f-0e1b-400d-8c8f-9f5804aa15ac nodeName:}" failed. No retries permitted until 2026-07-27 16:17:42.005268993 +0000 UTC m=+40.677803489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls") pod "dns-default-f7sqd" (UID: "76c82e0f-0e1b-400d-8c8f-9f5804aa15ac") : secret "dns-default-metrics-tls" not found Jul 27 16:17:41.505379 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.505357 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-tmp-dir\") pod \"dns-default-f7sqd\" (UID: \"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac\") " pod="openshift-dns/dns-default-f7sqd" Jul 27 16:17:41.505624 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.505604 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-config-volume\") pod \"dns-default-f7sqd\" (UID: \"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac\") " pod="openshift-dns/dns-default-f7sqd" Jul 27 16:17:41.509257 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.509232 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26b55f2f-e26a-4918-83b8-d49688d388fd-cert\") pod \"ingress-canary-dfvc2\" (UID: \"26b55f2f-e26a-4918-83b8-d49688d388fd\") " pod="openshift-ingress-canary/ingress-canary-dfvc2" Jul 27 16:17:41.515944 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.515922 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jfzl\" (UniqueName: \"kubernetes.io/projected/26b55f2f-e26a-4918-83b8-d49688d388fd-kube-api-access-7jfzl\") pod \"ingress-canary-dfvc2\" (UID: \"26b55f2f-e26a-4918-83b8-d49688d388fd\") " pod="openshift-ingress-canary/ingress-canary-dfvc2" Jul 27 16:17:41.516048 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.516000 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9xn5\" (UniqueName: \"kubernetes.io/projected/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-kube-api-access-r9xn5\") pod \"dns-default-f7sqd\" (UID: \"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac\") " pod="openshift-dns/dns-default-f7sqd" Jul 27 16:17:41.568071 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.568042 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dfvc2" Jul 27 16:17:41.780179 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.779877 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dfvc2"] Jul 27 16:17:41.784440 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:17:41.784417 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b55f2f_e26a_4918_83b8_d49688d388fd.slice/crio-c9ecebf26da18385e49d077a35126e5c4dcacdf414e0839f74f51f38f2037230 WatchSource:0}: Error finding container c9ecebf26da18385e49d077a35126e5c4dcacdf414e0839f74f51f38f2037230: Status 404 returned error can't find the container with id c9ecebf26da18385e49d077a35126e5c4dcacdf414e0839f74f51f38f2037230 Jul 27 16:17:41.844601 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.844570 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:17:41.844762 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.844687 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:17:41.844842 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.844821 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:17:41.847673 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.847655 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 16:17:41.847788 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.847700 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 16:17:41.848806 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.848790 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fft22\"" Jul 27 16:17:41.848942 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.848827 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 16:17:41.848942 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.848845 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 16:17:41.848942 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:41.848858 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9d9gv\"" Jul 27 16:17:42.007140 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:42.007098 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls\") pod \"dns-default-f7sqd\" (UID: \"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac\") " pod="openshift-dns/dns-default-f7sqd" Jul 27 16:17:42.007313 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:42.007245 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 16:17:42.007313 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:42.007305 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls podName:76c82e0f-0e1b-400d-8c8f-9f5804aa15ac nodeName:}" failed. No retries permitted until 2026-07-27 16:17:43.007289721 +0000 UTC m=+41.679824198 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls") pod "dns-default-f7sqd" (UID: "76c82e0f-0e1b-400d-8c8f-9f5804aa15ac") : secret "dns-default-metrics-tls" not found Jul 27 16:17:42.042917 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:42.042797 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dfvc2" event={"ID":"26b55f2f-e26a-4918-83b8-d49688d388fd","Type":"ContainerStarted","Data":"c9ecebf26da18385e49d077a35126e5c4dcacdf414e0839f74f51f38f2037230"} Jul 27 16:17:43.014951 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:43.014894 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls\") pod \"dns-default-f7sqd\" (UID: \"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac\") " pod="openshift-dns/dns-default-f7sqd" Jul 27 16:17:43.015620 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:43.015052 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 16:17:43.015620 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:43.015121 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls podName:76c82e0f-0e1b-400d-8c8f-9f5804aa15ac nodeName:}" failed. No retries permitted until 2026-07-27 16:17:45.015106607 +0000 UTC m=+43.687641085 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls") pod "dns-default-f7sqd" (UID: "76c82e0f-0e1b-400d-8c8f-9f5804aa15ac") : secret "dns-default-metrics-tls" not found Jul 27 16:17:45.030394 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:45.030359 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls\") pod \"dns-default-f7sqd\" (UID: \"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac\") " pod="openshift-dns/dns-default-f7sqd" Jul 27 16:17:45.030753 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:45.030531 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 16:17:45.030753 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:45.030592 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls podName:76c82e0f-0e1b-400d-8c8f-9f5804aa15ac nodeName:}" failed. No retries permitted until 2026-07-27 16:17:49.030577649 +0000 UTC m=+47.703112125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls") pod "dns-default-f7sqd" (UID: "76c82e0f-0e1b-400d-8c8f-9f5804aa15ac") : secret "dns-default-metrics-tls" not found Jul 27 16:17:45.051579 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:45.051538 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dfvc2" event={"ID":"26b55f2f-e26a-4918-83b8-d49688d388fd","Type":"ContainerStarted","Data":"ad43382fd975b24fd3776ce77f5b5100a596c6b558aef18e20a610fea5a8d4f1"} Jul 27 16:17:45.070807 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:45.070763 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dfvc2" podStartSLOduration=1.240482898 podStartE2EDuration="4.070750194s" totalImagesPullingTime="2.830267296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:17:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:45.069594633 +0000 UTC m=+43.742129136" watchObservedRunningTime="2026-07-27 16:17:45.070750194 +0000 UTC m=+43.743284692" Jul 27 16:17:49.059108 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:49.059059 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls\") pod \"dns-default-f7sqd\" (UID: \"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac\") " pod="openshift-dns/dns-default-f7sqd" Jul 27 16:17:49.059517 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:49.059183 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 16:17:49.059517 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:49.059266 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls podName:76c82e0f-0e1b-400d-8c8f-9f5804aa15ac nodeName:}" failed. No retries permitted until 2026-07-27 16:17:57.059250002 +0000 UTC m=+55.731784479 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls") pod "dns-default-f7sqd" (UID: "76c82e0f-0e1b-400d-8c8f-9f5804aa15ac") : secret "dns-default-metrics-tls" not found Jul 27 16:17:57.115678 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:17:57.115639 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls\") pod \"dns-default-f7sqd\" (UID: \"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac\") " pod="openshift-dns/dns-default-f7sqd" Jul 27 16:17:57.116243 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:57.115751 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 16:17:57.116243 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:17:57.115800 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls podName:76c82e0f-0e1b-400d-8c8f-9f5804aa15ac nodeName:}" failed. No retries permitted until 2026-07-27 16:18:13.11578648 +0000 UTC m=+71.788320957 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls") pod "dns-default-f7sqd" (UID: "76c82e0f-0e1b-400d-8c8f-9f5804aa15ac") : secret "dns-default-metrics-tls" not found Jul 27 16:18:07.481650 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:07.481598 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7bxt\" (UniqueName: \"kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt\") pod \"network-check-target-26l7z\" (UID: \"b1c1e78a-1087-4293-aca2-a34e14ee090b\") " pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:18:07.482075 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:07.481678 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs\") pod \"network-metrics-daemon-mg2dr\" (UID: \"fc3e4156-6cfc-430a-8842-a9d801cfb9f0\") " pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:18:07.484760 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:07.484736 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 16:18:07.484813 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:07.484786 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 16:18:07.492181 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:07.492159 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 27 16:18:07.492237 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:07.492220 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs podName:fc3e4156-6cfc-430a-8842-a9d801cfb9f0 nodeName:}" failed. No retries permitted until 2026-07-27 16:19:11.492203588 +0000 UTC m=+130.164738066 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs") pod "network-metrics-daemon-mg2dr" (UID: "fc3e4156-6cfc-430a-8842-a9d801cfb9f0") : secret "metrics-daemon-secret" not found Jul 27 16:18:07.494761 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:07.494747 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 16:18:07.506862 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:07.506838 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7bxt\" (UniqueName: \"kubernetes.io/projected/b1c1e78a-1087-4293-aca2-a34e14ee090b-kube-api-access-s7bxt\") pod \"network-check-target-26l7z\" (UID: \"b1c1e78a-1087-4293-aca2-a34e14ee090b\") " pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:18:07.582075 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:07.582039 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret\") pod \"global-pull-secret-syncer-lkclq\" (UID: \"efd59b32-c96c-4808-9a83-b24cdc2e3dc7\") " pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:18:07.584963 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:07.584945 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 16:18:07.594607 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:07.594589 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/efd59b32-c96c-4808-9a83-b24cdc2e3dc7-original-pull-secret\") pod \"global-pull-secret-syncer-lkclq\" (UID: \"efd59b32-c96c-4808-9a83-b24cdc2e3dc7\") " pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:18:07.655506 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:07.655471 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkclq" Jul 27 16:18:07.668125 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:07.668103 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fft22\"" Jul 27 16:18:07.675632 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:07.675613 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:18:07.789936 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:07.789888 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lkclq"] Jul 27 16:18:07.793005 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:18:07.792980 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd59b32_c96c_4808_9a83_b24cdc2e3dc7.slice/crio-e4d2cf4794789198d234e04845c23af14414b7f8ac167b45d5baee2894215a94 WatchSource:0}: Error finding container e4d2cf4794789198d234e04845c23af14414b7f8ac167b45d5baee2894215a94: Status 404 returned error can't find the container with id e4d2cf4794789198d234e04845c23af14414b7f8ac167b45d5baee2894215a94 Jul 27 16:18:07.807072 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:07.807051 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-26l7z"] Jul 27 16:18:07.810232 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:18:07.810211 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c1e78a_1087_4293_aca2_a34e14ee090b.slice/crio-d1ce3af94e60ee679eedebe3fe3d7b7d1bfd210c9ba394de55aefc4b3a5c5cc8 WatchSource:0}: Error finding container d1ce3af94e60ee679eedebe3fe3d7b7d1bfd210c9ba394de55aefc4b3a5c5cc8: Status 404 returned error can't find the container with id d1ce3af94e60ee679eedebe3fe3d7b7d1bfd210c9ba394de55aefc4b3a5c5cc8 Jul 27 16:18:08.043565 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:08.043541 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9gll" Jul 27 16:18:08.104603 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:08.104572 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-26l7z" event={"ID":"b1c1e78a-1087-4293-aca2-a34e14ee090b","Type":"ContainerStarted","Data":"d1ce3af94e60ee679eedebe3fe3d7b7d1bfd210c9ba394de55aefc4b3a5c5cc8"} Jul 27 16:18:08.105652 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:08.105630 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lkclq" event={"ID":"efd59b32-c96c-4808-9a83-b24cdc2e3dc7","Type":"ContainerStarted","Data":"e4d2cf4794789198d234e04845c23af14414b7f8ac167b45d5baee2894215a94"} Jul 27 16:18:11.115057 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:11.115021 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-26l7z" event={"ID":"b1c1e78a-1087-4293-aca2-a34e14ee090b","Type":"ContainerStarted","Data":"3dc2fd247cf3ecfab7b629aa75bec2595dddeb047ed8bda3b4c00fe3ae7f97d5"} Jul 27 16:18:11.115455 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:11.115176 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:18:11.140457 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:11.140407 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-26l7z" podStartSLOduration=66.267486644 podStartE2EDuration="1m9.14039353s" totalImagesPullingTime="2.872906886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:11.138442049 +0000 UTC m=+69.810976548" watchObservedRunningTime="2026-07-27 16:18:11.14039353 +0000 UTC m=+69.812928028" Jul 27 16:18:13.121972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:13.121939 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lkclq" event={"ID":"efd59b32-c96c-4808-9a83-b24cdc2e3dc7","Type":"ContainerStarted","Data":"5d44e8416662ed5235ff55ea0d3927f9d747a252b3eddb0f238fd39b255058c1"} Jul 27 16:18:13.122342 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:13.122265 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls\") pod \"dns-default-f7sqd\" (UID: \"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac\") " pod="openshift-dns/dns-default-f7sqd" Jul 27 16:18:13.122382 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:13.122366 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 16:18:13.122424 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:13.122412 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls podName:76c82e0f-0e1b-400d-8c8f-9f5804aa15ac nodeName:}" failed. No retries permitted until 2026-07-27 16:18:45.122399951 +0000 UTC m=+103.794934428 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls") pod "dns-default-f7sqd" (UID: "76c82e0f-0e1b-400d-8c8f-9f5804aa15ac") : secret "dns-default-metrics-tls" not found Jul 27 16:18:13.143617 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:13.143568 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lkclq" podStartSLOduration=65.27109733 podStartE2EDuration="1m10.143554147s" totalImagesPullingTime="4.872456817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:13.142831253 +0000 UTC m=+71.815365753" watchObservedRunningTime="2026-07-27 16:18:13.143554147 +0000 UTC m=+71.816088647" Jul 27 16:18:21.226023 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.225988 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-j4h8x"] Jul 27 16:18:21.227817 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.227798 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-j4h8x" Jul 27 16:18:21.230870 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.230850 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2"] Jul 27 16:18:21.231412 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.231383 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6pxnv\"" Jul 27 16:18:21.231412 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.231391 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 27 16:18:21.231553 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.231415 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 27 16:18:21.232752 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.232730 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2" Jul 27 16:18:21.236836 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.236819 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 27 16:18:21.236917 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.236839 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 27 16:18:21.238402 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.238384 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bkwx6\"" Jul 27 16:18:21.239110 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.239093 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 27 16:18:21.239264 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.239232 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 27 16:18:21.240981 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.240924 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-j4h8x"] Jul 27 16:18:21.248194 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.248175 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2"] Jul 27 16:18:21.280000 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.279970 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2528af38-f6e1-4c69-859a-6d00a5301ce4-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-4skw2\" (UID: \"2528af38-f6e1-4c69-859a-6d00a5301ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2" Jul 27 16:18:21.280000 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.280002 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2528af38-f6e1-4c69-859a-6d00a5301ce4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-4skw2\" (UID: \"2528af38-f6e1-4c69-859a-6d00a5301ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2" Jul 27 16:18:21.280204 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.280034 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8fjx\" (UniqueName: \"kubernetes.io/projected/2528af38-f6e1-4c69-859a-6d00a5301ce4-kube-api-access-p8fjx\") pod \"cluster-monitoring-operator-548b489cb8-4skw2\" (UID: \"2528af38-f6e1-4c69-859a-6d00a5301ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2" Jul 27 16:18:21.280204 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.280114 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2337956f-94b4-44b9-9404-7cc9de6e6b0f-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-j4h8x\" (UID: \"2337956f-94b4-44b9-9404-7cc9de6e6b0f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-j4h8x" Jul 27 16:18:21.280204 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.280141 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2337956f-94b4-44b9-9404-7cc9de6e6b0f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-j4h8x\" (UID: \"2337956f-94b4-44b9-9404-7cc9de6e6b0f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-j4h8x" Jul 27 16:18:21.332184 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.332153 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2"] Jul 27 16:18:21.334584 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.334568 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-dbfs5"] Jul 27 16:18:21.334705 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.334686 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2" Jul 27 16:18:21.336585 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.336568 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wsq68"] Jul 27 16:18:21.336715 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.336698 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.337848 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.337830 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 27 16:18:21.337978 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.337959 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 27 16:18:21.338140 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.338117 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tp7dv\"" Jul 27 16:18:21.338336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.338323 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 27 16:18:21.338568 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.338550 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wsq68" Jul 27 16:18:21.340177 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.340157 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 27 16:18:21.340267 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.340247 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 27 16:18:21.340267 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.340249 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 27 16:18:21.340388 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.340260 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nlxbd\"" Jul 27 16:18:21.340465 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.340446 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 27 16:18:21.341594 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.341572 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 27 16:18:21.342085 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.342068 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 27 16:18:21.342213 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.342190 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-mnvjr\"" Jul 27 16:18:21.342213 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.342091 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 27 16:18:21.342347 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.342128 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 27 16:18:21.345715 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.345694 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 27 16:18:21.347345 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.347324 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2"] Jul 27 16:18:21.348314 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.348139 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-dbfs5"] Jul 27 16:18:21.362963 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.362941 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wsq68"] Jul 27 16:18:21.381011 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.380976 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-597fq\" (UniqueName: \"kubernetes.io/projected/359a50cb-e66a-4a77-8ae8-33bc152ad58f-kube-api-access-597fq\") pod \"insights-operator-5885fdb759-dbfs5\" (UID: \"359a50cb-e66a-4a77-8ae8-33bc152ad58f\") " pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.381011 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.381017 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2337956f-94b4-44b9-9404-7cc9de6e6b0f-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-j4h8x\" (UID: \"2337956f-94b4-44b9-9404-7cc9de6e6b0f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-j4h8x" Jul 27 16:18:21.381145 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.381056 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2337956f-94b4-44b9-9404-7cc9de6e6b0f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-j4h8x\" (UID: \"2337956f-94b4-44b9-9404-7cc9de6e6b0f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-j4h8x" Jul 27 16:18:21.381145 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.381093 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-kgxh2\" (UID: \"0e60c2da-59d2-4c42-bb33-3d4c22b3b90c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2" Jul 27 16:18:21.381145 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.381125 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/359a50cb-e66a-4a77-8ae8-33bc152ad58f-tmp\") pod \"insights-operator-5885fdb759-dbfs5\" (UID: \"359a50cb-e66a-4a77-8ae8-33bc152ad58f\") " pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.381266 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:21.381168 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 16:18:21.381266 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.381171 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6-config\") pod \"service-ca-operator-5d6c75f84f-wsq68\" (UID: \"3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wsq68" Jul 27 16:18:21.381266 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.381195 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qm8p\" (UniqueName: \"kubernetes.io/projected/3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6-kube-api-access-6qm8p\") pod \"service-ca-operator-5d6c75f84f-wsq68\" (UID: \"3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wsq68" Jul 27 16:18:21.381266 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.381240 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqwk7\" (UniqueName: \"kubernetes.io/projected/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-kube-api-access-rqwk7\") pod \"cluster-samples-operator-6748489656-kgxh2\" (UID: \"0e60c2da-59d2-4c42-bb33-3d4c22b3b90c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2" Jul 27 16:18:21.381266 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:21.381264 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2337956f-94b4-44b9-9404-7cc9de6e6b0f-networking-console-plugin-cert podName:2337956f-94b4-44b9-9404-7cc9de6e6b0f nodeName:}" failed. No retries permitted until 2026-07-27 16:18:21.881249291 +0000 UTC m=+80.553783773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2337956f-94b4-44b9-9404-7cc9de6e6b0f-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-j4h8x" (UID: "2337956f-94b4-44b9-9404-7cc9de6e6b0f") : secret "networking-console-plugin-cert" not found Jul 27 16:18:21.381441 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.381300 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/359a50cb-e66a-4a77-8ae8-33bc152ad58f-snapshots\") pod \"insights-operator-5885fdb759-dbfs5\" (UID: \"359a50cb-e66a-4a77-8ae8-33bc152ad58f\") " pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.381441 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.381317 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/359a50cb-e66a-4a77-8ae8-33bc152ad58f-service-ca-bundle\") pod \"insights-operator-5885fdb759-dbfs5\" (UID: \"359a50cb-e66a-4a77-8ae8-33bc152ad58f\") " pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.381441 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.381344 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/359a50cb-e66a-4a77-8ae8-33bc152ad58f-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-dbfs5\" (UID: \"359a50cb-e66a-4a77-8ae8-33bc152ad58f\") " pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.381441 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.381379 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2528af38-f6e1-4c69-859a-6d00a5301ce4-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-4skw2\" (UID: \"2528af38-f6e1-4c69-859a-6d00a5301ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2" Jul 27 16:18:21.381441 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.381395 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2528af38-f6e1-4c69-859a-6d00a5301ce4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-4skw2\" (UID: \"2528af38-f6e1-4c69-859a-6d00a5301ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2" Jul 27 16:18:21.381597 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.381447 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8fjx\" (UniqueName: \"kubernetes.io/projected/2528af38-f6e1-4c69-859a-6d00a5301ce4-kube-api-access-p8fjx\") pod \"cluster-monitoring-operator-548b489cb8-4skw2\" (UID: \"2528af38-f6e1-4c69-859a-6d00a5301ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2" Jul 27 16:18:21.381597 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:21.381457 2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 16:18:21.381597 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.381474 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/359a50cb-e66a-4a77-8ae8-33bc152ad58f-serving-cert\") pod \"insights-operator-5885fdb759-dbfs5\" (UID: \"359a50cb-e66a-4a77-8ae8-33bc152ad58f\") " pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.381597 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:21.381498 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2528af38-f6e1-4c69-859a-6d00a5301ce4-cluster-monitoring-operator-tls podName:2528af38-f6e1-4c69-859a-6d00a5301ce4 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:21.881485777 +0000 UTC m=+80.554020275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2528af38-f6e1-4c69-859a-6d00a5301ce4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-4skw2" (UID: "2528af38-f6e1-4c69-859a-6d00a5301ce4") : secret "cluster-monitoring-operator-tls" not found Jul 27 16:18:21.381597 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.381528 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6-serving-cert\") pod \"service-ca-operator-5d6c75f84f-wsq68\" (UID: \"3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wsq68" Jul 27 16:18:21.381597 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.381587 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2337956f-94b4-44b9-9404-7cc9de6e6b0f-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-j4h8x\" (UID: \"2337956f-94b4-44b9-9404-7cc9de6e6b0f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-j4h8x" Jul 27 16:18:21.382012 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.381992 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2528af38-f6e1-4c69-859a-6d00a5301ce4-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-4skw2\" (UID: \"2528af38-f6e1-4c69-859a-6d00a5301ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2" Jul 27 16:18:21.394775 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.394744 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8fjx\" (UniqueName: \"kubernetes.io/projected/2528af38-f6e1-4c69-859a-6d00a5301ce4-kube-api-access-p8fjx\") pod \"cluster-monitoring-operator-548b489cb8-4skw2\" (UID: \"2528af38-f6e1-4c69-859a-6d00a5301ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2" Jul 27 16:18:21.481875 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.481788 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-kgxh2\" (UID: \"0e60c2da-59d2-4c42-bb33-3d4c22b3b90c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2" Jul 27 16:18:21.481875 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.481826 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/359a50cb-e66a-4a77-8ae8-33bc152ad58f-tmp\") pod \"insights-operator-5885fdb759-dbfs5\" (UID: \"359a50cb-e66a-4a77-8ae8-33bc152ad58f\") " pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.482061 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:21.481945 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 16:18:21.482061 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.481955 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6-config\") pod \"service-ca-operator-5d6c75f84f-wsq68\" (UID: \"3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wsq68" Jul 27 16:18:21.482061 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.481989 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qm8p\" (UniqueName: \"kubernetes.io/projected/3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6-kube-api-access-6qm8p\") pod \"service-ca-operator-5d6c75f84f-wsq68\" (UID: \"3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wsq68" Jul 27 16:18:21.482061 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:21.482013 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-samples-operator-tls podName:0e60c2da-59d2-4c42-bb33-3d4c22b3b90c nodeName:}" failed. No retries permitted until 2026-07-27 16:18:21.981996539 +0000 UTC m=+80.654531019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-samples-operator-tls") pod "cluster-samples-operator-6748489656-kgxh2" (UID: "0e60c2da-59d2-4c42-bb33-3d4c22b3b90c") : secret "samples-operator-tls" not found Jul 27 16:18:21.482226 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.482044 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqwk7\" (UniqueName: \"kubernetes.io/projected/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-kube-api-access-rqwk7\") pod \"cluster-samples-operator-6748489656-kgxh2\" (UID: \"0e60c2da-59d2-4c42-bb33-3d4c22b3b90c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2" Jul 27 16:18:21.482226 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.482091 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/359a50cb-e66a-4a77-8ae8-33bc152ad58f-snapshots\") pod \"insights-operator-5885fdb759-dbfs5\" (UID: \"359a50cb-e66a-4a77-8ae8-33bc152ad58f\") " pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.482226 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.482114 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/359a50cb-e66a-4a77-8ae8-33bc152ad58f-service-ca-bundle\") pod \"insights-operator-5885fdb759-dbfs5\" (UID: \"359a50cb-e66a-4a77-8ae8-33bc152ad58f\") " pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.482226 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.482150 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/359a50cb-e66a-4a77-8ae8-33bc152ad58f-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-dbfs5\" (UID: \"359a50cb-e66a-4a77-8ae8-33bc152ad58f\") " pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.482226 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.482196 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/359a50cb-e66a-4a77-8ae8-33bc152ad58f-tmp\") pod \"insights-operator-5885fdb759-dbfs5\" (UID: \"359a50cb-e66a-4a77-8ae8-33bc152ad58f\") " pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.482427 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.482251 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/359a50cb-e66a-4a77-8ae8-33bc152ad58f-serving-cert\") pod \"insights-operator-5885fdb759-dbfs5\" (UID: \"359a50cb-e66a-4a77-8ae8-33bc152ad58f\") " pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.482427 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.482291 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6-serving-cert\") pod \"service-ca-operator-5d6c75f84f-wsq68\" (UID: \"3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wsq68" Jul 27 16:18:21.482427 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.482318 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-597fq\" (UniqueName: \"kubernetes.io/projected/359a50cb-e66a-4a77-8ae8-33bc152ad58f-kube-api-access-597fq\") pod \"insights-operator-5885fdb759-dbfs5\" (UID: \"359a50cb-e66a-4a77-8ae8-33bc152ad58f\") " pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.482646 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.482624 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6-config\") pod \"service-ca-operator-5d6c75f84f-wsq68\" (UID: \"3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wsq68" Jul 27 16:18:21.482704 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.482648 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/359a50cb-e66a-4a77-8ae8-33bc152ad58f-service-ca-bundle\") pod \"insights-operator-5885fdb759-dbfs5\" (UID: \"359a50cb-e66a-4a77-8ae8-33bc152ad58f\") " pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.482770 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.482748 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/359a50cb-e66a-4a77-8ae8-33bc152ad58f-snapshots\") pod \"insights-operator-5885fdb759-dbfs5\" (UID: \"359a50cb-e66a-4a77-8ae8-33bc152ad58f\") " pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.483218 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.483198 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/359a50cb-e66a-4a77-8ae8-33bc152ad58f-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-dbfs5\" (UID: \"359a50cb-e66a-4a77-8ae8-33bc152ad58f\") " pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.484672 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.484655 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6-serving-cert\") pod \"service-ca-operator-5d6c75f84f-wsq68\" (UID: \"3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wsq68" Jul 27 16:18:21.484719 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.484701 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/359a50cb-e66a-4a77-8ae8-33bc152ad58f-serving-cert\") pod \"insights-operator-5885fdb759-dbfs5\" (UID: \"359a50cb-e66a-4a77-8ae8-33bc152ad58f\") " pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.491741 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.491719 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqwk7\" (UniqueName: \"kubernetes.io/projected/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-kube-api-access-rqwk7\") pod \"cluster-samples-operator-6748489656-kgxh2\" (UID: \"0e60c2da-59d2-4c42-bb33-3d4c22b3b90c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2" Jul 27 16:18:21.491741 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.491731 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-597fq\" (UniqueName: \"kubernetes.io/projected/359a50cb-e66a-4a77-8ae8-33bc152ad58f-kube-api-access-597fq\") pod \"insights-operator-5885fdb759-dbfs5\" (UID: \"359a50cb-e66a-4a77-8ae8-33bc152ad58f\") " pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.492522 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.492500 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qm8p\" (UniqueName: \"kubernetes.io/projected/3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6-kube-api-access-6qm8p\") pod \"service-ca-operator-5d6c75f84f-wsq68\" (UID: \"3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wsq68" Jul 27 16:18:21.652095 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.652062 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-dbfs5" Jul 27 16:18:21.657177 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.657157 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wsq68" Jul 27 16:18:21.784630 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.784575 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wsq68"] Jul 27 16:18:21.787461 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:18:21.787435 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3043ca1d_5bc5_42d3_8dcb_7a9211ba39d6.slice/crio-470170d9cd9a922cef8df4cf3e337f9b4817667ac617bec566bd79d94d181622 WatchSource:0}: Error finding container 470170d9cd9a922cef8df4cf3e337f9b4817667ac617bec566bd79d94d181622: Status 404 returned error can't find the container with id 470170d9cd9a922cef8df4cf3e337f9b4817667ac617bec566bd79d94d181622 Jul 27 16:18:21.803857 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.803829 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-dbfs5"] Jul 27 16:18:21.807567 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:18:21.807539 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359a50cb_e66a_4a77_8ae8_33bc152ad58f.slice/crio-d1daf1436f44dbd8c6d3bf3b8f89de5998fdea6a67439c67b069f7aa17065226 WatchSource:0}: Error finding container d1daf1436f44dbd8c6d3bf3b8f89de5998fdea6a67439c67b069f7aa17065226: Status 404 returned error can't find the container with id d1daf1436f44dbd8c6d3bf3b8f89de5998fdea6a67439c67b069f7aa17065226 Jul 27 16:18:21.884779 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.884750 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2528af38-f6e1-4c69-859a-6d00a5301ce4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-4skw2\" (UID: \"2528af38-f6e1-4c69-859a-6d00a5301ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2" Jul 27 16:18:21.884999 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.884800 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2337956f-94b4-44b9-9404-7cc9de6e6b0f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-j4h8x\" (UID: \"2337956f-94b4-44b9-9404-7cc9de6e6b0f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-j4h8x" Jul 27 16:18:21.884999 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:21.884921 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 16:18:21.884999 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:21.884925 2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 16:18:21.884999 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:21.884968 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2337956f-94b4-44b9-9404-7cc9de6e6b0f-networking-console-plugin-cert podName:2337956f-94b4-44b9-9404-7cc9de6e6b0f nodeName:}" failed. No retries permitted until 2026-07-27 16:18:22.884955584 +0000 UTC m=+81.557490060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2337956f-94b4-44b9-9404-7cc9de6e6b0f-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-j4h8x" (UID: "2337956f-94b4-44b9-9404-7cc9de6e6b0f") : secret "networking-console-plugin-cert" not found Jul 27 16:18:21.884999 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:21.884981 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2528af38-f6e1-4c69-859a-6d00a5301ce4-cluster-monitoring-operator-tls podName:2528af38-f6e1-4c69-859a-6d00a5301ce4 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:22.884974998 +0000 UTC m=+81.557509475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2528af38-f6e1-4c69-859a-6d00a5301ce4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-4skw2" (UID: "2528af38-f6e1-4c69-859a-6d00a5301ce4") : secret "cluster-monitoring-operator-tls" not found Jul 27 16:18:21.985505 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:21.985465 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-kgxh2\" (UID: \"0e60c2da-59d2-4c42-bb33-3d4c22b3b90c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2" Jul 27 16:18:21.985685 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:21.985621 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 16:18:21.985734 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:21.985686 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-samples-operator-tls podName:0e60c2da-59d2-4c42-bb33-3d4c22b3b90c nodeName:}" failed. No retries permitted until 2026-07-27 16:18:22.985669813 +0000 UTC m=+81.658204291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-samples-operator-tls") pod "cluster-samples-operator-6748489656-kgxh2" (UID: "0e60c2da-59d2-4c42-bb33-3d4c22b3b90c") : secret "samples-operator-tls" not found Jul 27 16:18:22.149252 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:22.149148 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-dbfs5" event={"ID":"359a50cb-e66a-4a77-8ae8-33bc152ad58f","Type":"ContainerStarted","Data":"d1daf1436f44dbd8c6d3bf3b8f89de5998fdea6a67439c67b069f7aa17065226"} Jul 27 16:18:22.150246 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:22.150222 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wsq68" event={"ID":"3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6","Type":"ContainerStarted","Data":"470170d9cd9a922cef8df4cf3e337f9b4817667ac617bec566bd79d94d181622"} Jul 27 16:18:22.891851 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:22.891811 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2337956f-94b4-44b9-9404-7cc9de6e6b0f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-j4h8x\" (UID: \"2337956f-94b4-44b9-9404-7cc9de6e6b0f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-j4h8x" Jul 27 16:18:22.892335 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:22.891941 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2528af38-f6e1-4c69-859a-6d00a5301ce4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-4skw2\" (UID: \"2528af38-f6e1-4c69-859a-6d00a5301ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2" Jul 27 16:18:22.892335 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:22.891994 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 16:18:22.892335 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:22.892061 2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 16:18:22.892335 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:22.892082 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2337956f-94b4-44b9-9404-7cc9de6e6b0f-networking-console-plugin-cert podName:2337956f-94b4-44b9-9404-7cc9de6e6b0f nodeName:}" failed. No retries permitted until 2026-07-27 16:18:24.892057845 +0000 UTC m=+83.564592338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2337956f-94b4-44b9-9404-7cc9de6e6b0f-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-j4h8x" (UID: "2337956f-94b4-44b9-9404-7cc9de6e6b0f") : secret "networking-console-plugin-cert" not found Jul 27 16:18:22.892335 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:22.892109 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2528af38-f6e1-4c69-859a-6d00a5301ce4-cluster-monitoring-operator-tls podName:2528af38-f6e1-4c69-859a-6d00a5301ce4 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:24.892093213 +0000 UTC m=+83.564627708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2528af38-f6e1-4c69-859a-6d00a5301ce4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-4skw2" (UID: "2528af38-f6e1-4c69-859a-6d00a5301ce4") : secret "cluster-monitoring-operator-tls" not found Jul 27 16:18:22.992679 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:22.992646 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-kgxh2\" (UID: \"0e60c2da-59d2-4c42-bb33-3d4c22b3b90c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2" Jul 27 16:18:22.992933 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:22.992894 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 16:18:22.993053 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:22.993037 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-samples-operator-tls podName:0e60c2da-59d2-4c42-bb33-3d4c22b3b90c nodeName:}" failed. No retries permitted until 2026-07-27 16:18:24.993014766 +0000 UTC m=+83.665549262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-samples-operator-tls") pod "cluster-samples-operator-6748489656-kgxh2" (UID: "0e60c2da-59d2-4c42-bb33-3d4c22b3b90c") : secret "samples-operator-tls" not found Jul 27 16:18:24.908219 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:24.908182 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2528af38-f6e1-4c69-859a-6d00a5301ce4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-4skw2\" (UID: \"2528af38-f6e1-4c69-859a-6d00a5301ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2" Jul 27 16:18:24.908614 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:24.908258 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2337956f-94b4-44b9-9404-7cc9de6e6b0f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-j4h8x\" (UID: \"2337956f-94b4-44b9-9404-7cc9de6e6b0f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-j4h8x" Jul 27 16:18:24.908614 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:24.908333 2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 16:18:24.908614 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:24.908393 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2528af38-f6e1-4c69-859a-6d00a5301ce4-cluster-monitoring-operator-tls podName:2528af38-f6e1-4c69-859a-6d00a5301ce4 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:28.908380337 +0000 UTC m=+87.580914814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2528af38-f6e1-4c69-859a-6d00a5301ce4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-4skw2" (UID: "2528af38-f6e1-4c69-859a-6d00a5301ce4") : secret "cluster-monitoring-operator-tls" not found Jul 27 16:18:24.908614 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:24.908419 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 16:18:24.908614 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:24.908484 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2337956f-94b4-44b9-9404-7cc9de6e6b0f-networking-console-plugin-cert podName:2337956f-94b4-44b9-9404-7cc9de6e6b0f nodeName:}" failed. No retries permitted until 2026-07-27 16:18:28.908466459 +0000 UTC m=+87.581000939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2337956f-94b4-44b9-9404-7cc9de6e6b0f-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-j4h8x" (UID: "2337956f-94b4-44b9-9404-7cc9de6e6b0f") : secret "networking-console-plugin-cert" not found Jul 27 16:18:25.009395 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:25.009360 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-kgxh2\" (UID: \"0e60c2da-59d2-4c42-bb33-3d4c22b3b90c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2" Jul 27 16:18:25.009569 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:25.009498 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 16:18:25.009569 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:25.009554 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-samples-operator-tls podName:0e60c2da-59d2-4c42-bb33-3d4c22b3b90c nodeName:}" failed. No retries permitted until 2026-07-27 16:18:29.009540816 +0000 UTC m=+87.682075293 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-samples-operator-tls") pod "cluster-samples-operator-6748489656-kgxh2" (UID: "0e60c2da-59d2-4c42-bb33-3d4c22b3b90c") : secret "samples-operator-tls" not found Jul 27 16:18:25.161034 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:25.160939 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wsq68" event={"ID":"3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6","Type":"ContainerStarted","Data":"0d97df4fba303c5c5f50d8ad5d5487a06664f985ca911aa77adacdbbe3e0fe88"} Jul 27 16:18:25.162540 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:25.162513 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-dbfs5" event={"ID":"359a50cb-e66a-4a77-8ae8-33bc152ad58f","Type":"ContainerStarted","Data":"4f3b52b2b6ce7ed8c722d1f1e9173245b56b894bc610526ae517609c1b9f3297"} Jul 27 16:18:25.196773 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:25.196712 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wsq68" podStartSLOduration=1.594231279 podStartE2EDuration="4.196693167s" totalImagesPullingTime="2.602461888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:25.195809473 +0000 UTC m=+83.868343972" watchObservedRunningTime="2026-07-27 16:18:25.196693167 +0000 UTC m=+83.869227663" Jul 27 16:18:25.231830 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:25.231777 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-dbfs5" podStartSLOduration=1.651216574 podStartE2EDuration="4.231761951s" totalImagesPullingTime="2.580545377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:25.230618041 +0000 UTC m=+83.903152551" watchObservedRunningTime="2026-07-27 16:18:25.231761951 +0000 UTC m=+83.904296450" Jul 27 16:18:26.427195 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.427160 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-pplnk"] Jul 27 16:18:26.430173 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.430156 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-pplnk" Jul 27 16:18:26.433431 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.433402 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 27 16:18:26.433943 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.433928 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 27 16:18:26.434797 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.434782 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-clqn7\"" Jul 27 16:18:26.444806 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.444783 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-pplnk"] Jul 27 16:18:26.521215 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.521175 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7pnt\" (UniqueName: \"kubernetes.io/projected/d64bf603-43e5-434a-9a73-b7bc690f329b-kube-api-access-d7pnt\") pod \"migrator-6865c9967-pplnk\" (UID: \"d64bf603-43e5-434a-9a73-b7bc690f329b\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-pplnk" Jul 27 16:18:26.622581 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.622543 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7pnt\" (UniqueName: \"kubernetes.io/projected/d64bf603-43e5-434a-9a73-b7bc690f329b-kube-api-access-d7pnt\") pod \"migrator-6865c9967-pplnk\" (UID: \"d64bf603-43e5-434a-9a73-b7bc690f329b\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-pplnk" Jul 27 16:18:26.635283 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.635259 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7pnt\" (UniqueName: \"kubernetes.io/projected/d64bf603-43e5-434a-9a73-b7bc690f329b-kube-api-access-d7pnt\") pod \"migrator-6865c9967-pplnk\" (UID: \"d64bf603-43e5-434a-9a73-b7bc690f329b\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-pplnk" Jul 27 16:18:26.739083 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.739005 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-pplnk" Jul 27 16:18:26.783182 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.783150 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-68vxh"] Jul 27 16:18:26.786625 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.785757 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:26.790504 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.790479 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m7sx7\"" Jul 27 16:18:26.791657 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.791638 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 27 16:18:26.792468 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.792166 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 27 16:18:26.825161 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.823836 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba81a68e-6c72-4447-ad53-a81f29cee2a9-crio-socket\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:26.825421 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.825398 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:26.825534 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.825519 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gttfg\" (UniqueName: \"kubernetes.io/projected/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-api-access-gttfg\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:26.825684 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.825668 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba81a68e-6c72-4447-ad53-a81f29cee2a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:26.825781 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.825769 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba81a68e-6c72-4447-ad53-a81f29cee2a9-data-volume\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:26.840772 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.840735 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-68vxh"] Jul 27 16:18:26.900009 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.899978 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-pplnk"] Jul 27 16:18:26.903997 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:18:26.903947 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64bf603_43e5_434a_9a73_b7bc690f329b.slice/crio-dbc701d7b21530d8f289e0becc098f4cb6dedc78185f1e9295f58ee73b53e317 WatchSource:0}: Error finding container dbc701d7b21530d8f289e0becc098f4cb6dedc78185f1e9295f58ee73b53e317: Status 404 returned error can't find the container with id dbc701d7b21530d8f289e0becc098f4cb6dedc78185f1e9295f58ee73b53e317 Jul 27 16:18:26.927111 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.927082 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:26.927209 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.927114 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gttfg\" (UniqueName: \"kubernetes.io/projected/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-api-access-gttfg\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:26.927209 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.927170 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba81a68e-6c72-4447-ad53-a81f29cee2a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:26.927209 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.927187 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba81a68e-6c72-4447-ad53-a81f29cee2a9-data-volume\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:26.927209 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:26.927204 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 16:18:26.927383 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.927232 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba81a68e-6c72-4447-ad53-a81f29cee2a9-crio-socket\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:26.927383 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:26.927268 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-rbac-proxy-cm podName:ba81a68e-6c72-4447-ad53-a81f29cee2a9 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:27.42725314 +0000 UTC m=+86.099787617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-68vxh" (UID: "ba81a68e-6c72-4447-ad53-a81f29cee2a9") : configmap "kube-rbac-proxy" not found Jul 27 16:18:26.927383 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.927309 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba81a68e-6c72-4447-ad53-a81f29cee2a9-crio-socket\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:26.927383 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:26.927326 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 16:18:26.927383 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:26.927376 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba81a68e-6c72-4447-ad53-a81f29cee2a9-insights-runtime-extractor-tls podName:ba81a68e-6c72-4447-ad53-a81f29cee2a9 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:27.427361163 +0000 UTC m=+86.099895646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ba81a68e-6c72-4447-ad53-a81f29cee2a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-68vxh" (UID: "ba81a68e-6c72-4447-ad53-a81f29cee2a9") : secret "insights-runtime-extractor-tls" not found Jul 27 16:18:26.927638 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.927592 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba81a68e-6c72-4447-ad53-a81f29cee2a9-data-volume\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:26.944275 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:26.944249 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gttfg\" (UniqueName: \"kubernetes.io/projected/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-api-access-gttfg\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:27.171267 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.171230 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-pplnk" event={"ID":"d64bf603-43e5-434a-9a73-b7bc690f329b","Type":"ContainerStarted","Data":"dbc701d7b21530d8f289e0becc098f4cb6dedc78185f1e9295f58ee73b53e317"} Jul 27 16:18:27.431576 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.431488 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:27.431576 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.431560 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba81a68e-6c72-4447-ad53-a81f29cee2a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:27.432033 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:27.431641 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 16:18:27.432033 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:27.431674 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 16:18:27.432033 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:27.431721 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-rbac-proxy-cm podName:ba81a68e-6c72-4447-ad53-a81f29cee2a9 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:28.431703008 +0000 UTC m=+87.104237486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-68vxh" (UID: "ba81a68e-6c72-4447-ad53-a81f29cee2a9") : configmap "kube-rbac-proxy" not found Jul 27 16:18:27.432033 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:27.431741 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba81a68e-6c72-4447-ad53-a81f29cee2a9-insights-runtime-extractor-tls podName:ba81a68e-6c72-4447-ad53-a81f29cee2a9 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:28.431731878 +0000 UTC m=+87.104266362 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ba81a68e-6c72-4447-ad53-a81f29cee2a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-68vxh" (UID: "ba81a68e-6c72-4447-ad53-a81f29cee2a9") : secret "insights-runtime-extractor-tls" not found Jul 27 16:18:27.647768 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.647732 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-j4m7l"] Jul 27 16:18:27.649956 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.649941 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-j4m7l" Jul 27 16:18:27.653751 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.653730 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 27 16:18:27.653956 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.653844 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 27 16:18:27.654044 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.653967 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 27 16:18:27.655088 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.655068 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cjxbt\"" Jul 27 16:18:27.655203 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.655133 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 27 16:18:27.661955 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.661934 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-j4m7l"] Jul 27 16:18:27.734252 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.734122 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d93f123-6e30-498b-9e34-b1f91ab2f51b-signing-key\") pod \"service-ca-658d7b7c78-j4m7l\" (UID: \"1d93f123-6e30-498b-9e34-b1f91ab2f51b\") " pod="openshift-service-ca/service-ca-658d7b7c78-j4m7l" Jul 27 16:18:27.734252 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.734251 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d93f123-6e30-498b-9e34-b1f91ab2f51b-signing-cabundle\") pod \"service-ca-658d7b7c78-j4m7l\" (UID: \"1d93f123-6e30-498b-9e34-b1f91ab2f51b\") " pod="openshift-service-ca/service-ca-658d7b7c78-j4m7l" Jul 27 16:18:27.734464 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.734322 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5gcj\" (UniqueName: \"kubernetes.io/projected/1d93f123-6e30-498b-9e34-b1f91ab2f51b-kube-api-access-t5gcj\") pod \"service-ca-658d7b7c78-j4m7l\" (UID: \"1d93f123-6e30-498b-9e34-b1f91ab2f51b\") " pod="openshift-service-ca/service-ca-658d7b7c78-j4m7l" Jul 27 16:18:27.835774 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.835745 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d93f123-6e30-498b-9e34-b1f91ab2f51b-signing-key\") pod \"service-ca-658d7b7c78-j4m7l\" (UID: \"1d93f123-6e30-498b-9e34-b1f91ab2f51b\") " pod="openshift-service-ca/service-ca-658d7b7c78-j4m7l" Jul 27 16:18:27.835933 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.835784 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d93f123-6e30-498b-9e34-b1f91ab2f51b-signing-cabundle\") pod \"service-ca-658d7b7c78-j4m7l\" (UID: \"1d93f123-6e30-498b-9e34-b1f91ab2f51b\") " pod="openshift-service-ca/service-ca-658d7b7c78-j4m7l" Jul 27 16:18:27.835933 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.835838 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5gcj\" (UniqueName: \"kubernetes.io/projected/1d93f123-6e30-498b-9e34-b1f91ab2f51b-kube-api-access-t5gcj\") pod \"service-ca-658d7b7c78-j4m7l\" (UID: \"1d93f123-6e30-498b-9e34-b1f91ab2f51b\") " pod="openshift-service-ca/service-ca-658d7b7c78-j4m7l" Jul 27 16:18:27.836601 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.836576 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d93f123-6e30-498b-9e34-b1f91ab2f51b-signing-cabundle\") pod \"service-ca-658d7b7c78-j4m7l\" (UID: \"1d93f123-6e30-498b-9e34-b1f91ab2f51b\") " pod="openshift-service-ca/service-ca-658d7b7c78-j4m7l" Jul 27 16:18:27.838410 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.838388 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d93f123-6e30-498b-9e34-b1f91ab2f51b-signing-key\") pod \"service-ca-658d7b7c78-j4m7l\" (UID: \"1d93f123-6e30-498b-9e34-b1f91ab2f51b\") " pod="openshift-service-ca/service-ca-658d7b7c78-j4m7l" Jul 27 16:18:27.852697 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.852669 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5gcj\" (UniqueName: \"kubernetes.io/projected/1d93f123-6e30-498b-9e34-b1f91ab2f51b-kube-api-access-t5gcj\") pod \"service-ca-658d7b7c78-j4m7l\" (UID: \"1d93f123-6e30-498b-9e34-b1f91ab2f51b\") " pod="openshift-service-ca/service-ca-658d7b7c78-j4m7l" Jul 27 16:18:27.958953 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:27.958914 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-j4m7l" Jul 27 16:18:28.157267 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:28.157239 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-j4m7l"] Jul 27 16:18:28.161104 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:18:28.161082 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d93f123_6e30_498b_9e34_b1f91ab2f51b.slice/crio-f74ed0ed7598b1f5670b32b2fa18cf690d111648dd32ddd2fbc968f25f3ee755 WatchSource:0}: Error finding container f74ed0ed7598b1f5670b32b2fa18cf690d111648dd32ddd2fbc968f25f3ee755: Status 404 returned error can't find the container with id f74ed0ed7598b1f5670b32b2fa18cf690d111648dd32ddd2fbc968f25f3ee755 Jul 27 16:18:28.176366 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:28.176342 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-pplnk" event={"ID":"d64bf603-43e5-434a-9a73-b7bc690f329b","Type":"ContainerStarted","Data":"bd686f32aa327bd4c1dadddb827bbcd309e55abb7b629030261d3701f6d10037"} Jul 27 16:18:28.177996 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:28.177965 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-j4m7l" event={"ID":"1d93f123-6e30-498b-9e34-b1f91ab2f51b","Type":"ContainerStarted","Data":"f74ed0ed7598b1f5670b32b2fa18cf690d111648dd32ddd2fbc968f25f3ee755"} Jul 27 16:18:28.441565 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:28.441474 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:28.441961 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:28.441566 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba81a68e-6c72-4447-ad53-a81f29cee2a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:28.441961 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:28.441590 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 16:18:28.441961 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:28.441657 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-rbac-proxy-cm podName:ba81a68e-6c72-4447-ad53-a81f29cee2a9 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:30.441642535 +0000 UTC m=+89.114177013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-68vxh" (UID: "ba81a68e-6c72-4447-ad53-a81f29cee2a9") : configmap "kube-rbac-proxy" not found Jul 27 16:18:28.441961 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:28.441673 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 16:18:28.441961 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:28.441723 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba81a68e-6c72-4447-ad53-a81f29cee2a9-insights-runtime-extractor-tls podName:ba81a68e-6c72-4447-ad53-a81f29cee2a9 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:30.441708062 +0000 UTC m=+89.114242544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ba81a68e-6c72-4447-ad53-a81f29cee2a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-68vxh" (UID: "ba81a68e-6c72-4447-ad53-a81f29cee2a9") : secret "insights-runtime-extractor-tls" not found Jul 27 16:18:28.946011 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:28.945951 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2528af38-f6e1-4c69-859a-6d00a5301ce4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-4skw2\" (UID: \"2528af38-f6e1-4c69-859a-6d00a5301ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2" Jul 27 16:18:28.946176 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:28.946029 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2337956f-94b4-44b9-9404-7cc9de6e6b0f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-j4h8x\" (UID: \"2337956f-94b4-44b9-9404-7cc9de6e6b0f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-j4h8x" Jul 27 16:18:28.946176 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:28.946140 2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 16:18:28.946254 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:28.946191 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 16:18:28.946254 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:28.946229 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2528af38-f6e1-4c69-859a-6d00a5301ce4-cluster-monitoring-operator-tls podName:2528af38-f6e1-4c69-859a-6d00a5301ce4 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:36.946208474 +0000 UTC m=+95.618742957 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2528af38-f6e1-4c69-859a-6d00a5301ce4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-4skw2" (UID: "2528af38-f6e1-4c69-859a-6d00a5301ce4") : secret "cluster-monitoring-operator-tls" not found Jul 27 16:18:28.946254 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:28.946249 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2337956f-94b4-44b9-9404-7cc9de6e6b0f-networking-console-plugin-cert podName:2337956f-94b4-44b9-9404-7cc9de6e6b0f nodeName:}" failed. No retries permitted until 2026-07-27 16:18:36.946239473 +0000 UTC m=+95.618773959 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2337956f-94b4-44b9-9404-7cc9de6e6b0f-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-j4h8x" (UID: "2337956f-94b4-44b9-9404-7cc9de6e6b0f") : secret "networking-console-plugin-cert" not found Jul 27 16:18:29.046791 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:29.046756 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-kgxh2\" (UID: \"0e60c2da-59d2-4c42-bb33-3d4c22b3b90c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2" Jul 27 16:18:29.046964 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:29.046942 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 16:18:29.047026 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:29.047013 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-samples-operator-tls podName:0e60c2da-59d2-4c42-bb33-3d4c22b3b90c nodeName:}" failed. No retries permitted until 2026-07-27 16:18:37.046993385 +0000 UTC m=+95.719527879 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-samples-operator-tls") pod "cluster-samples-operator-6748489656-kgxh2" (UID: "0e60c2da-59d2-4c42-bb33-3d4c22b3b90c") : secret "samples-operator-tls" not found Jul 27 16:18:29.182782 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:29.182735 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-pplnk" event={"ID":"d64bf603-43e5-434a-9a73-b7bc690f329b","Type":"ContainerStarted","Data":"955a3d20f0266f838ed7717bb38b5a0f6a0940279efe23d559d6089d6b7450bb"} Jul 27 16:18:29.184183 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:29.184153 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-j4m7l" event={"ID":"1d93f123-6e30-498b-9e34-b1f91ab2f51b","Type":"ContainerStarted","Data":"952a95a57366c5f8b83514517c12dbaf22920425166bb53d9dbfe22237f20841"} Jul 27 16:18:29.216986 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:29.216872 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-pplnk" podStartSLOduration=2.076903275 podStartE2EDuration="3.216857352s" totalImagesPullingTime="1.139954077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:29.216295984 +0000 UTC m=+87.888830484" watchObservedRunningTime="2026-07-27 16:18:29.216857352 +0000 UTC m=+87.889391850" Jul 27 16:18:29.307862 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:29.307835 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9fnfd_92a27900-e3d3-44c2-a7da-7c3ff58b7cfc/dns-node-resolver/0.log" Jul 27 16:18:30.458841 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:30.458803 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:30.459333 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:30.458891 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba81a68e-6c72-4447-ad53-a81f29cee2a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:30.459333 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:30.458940 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 16:18:30.459333 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:30.459003 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-rbac-proxy-cm podName:ba81a68e-6c72-4447-ad53-a81f29cee2a9 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:34.458988225 +0000 UTC m=+93.131522703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-68vxh" (UID: "ba81a68e-6c72-4447-ad53-a81f29cee2a9") : configmap "kube-rbac-proxy" not found Jul 27 16:18:30.459333 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:30.459048 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 16:18:30.459333 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:30.459099 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba81a68e-6c72-4447-ad53-a81f29cee2a9-insights-runtime-extractor-tls podName:ba81a68e-6c72-4447-ad53-a81f29cee2a9 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:34.459085856 +0000 UTC m=+93.131620335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ba81a68e-6c72-4447-ad53-a81f29cee2a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-68vxh" (UID: "ba81a68e-6c72-4447-ad53-a81f29cee2a9") : secret "insights-runtime-extractor-tls" not found Jul 27 16:18:30.705100 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:30.705072 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tnb6j_9b8bc6ae-9d34-41e4-8079-9149a29e6627/node-ca/0.log" Jul 27 16:18:31.109074 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:31.109040 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dfvc2_26b55f2f-e26a-4918-83b8-d49688d388fd/serve-healthcheck-canary/0.log" Jul 27 16:18:31.720179 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:31.720153 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-pplnk_d64bf603-43e5-434a-9a73-b7bc690f329b/migrator/0.log" Jul 27 16:18:31.905299 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:31.905273 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-pplnk_d64bf603-43e5-434a-9a73-b7bc690f329b/graceful-termination/0.log" Jul 27 16:18:34.490477 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:34.490438 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:34.490946 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:34.490510 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba81a68e-6c72-4447-ad53-a81f29cee2a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:34.490946 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:34.490559 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 16:18:34.490946 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:34.490619 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 16:18:34.490946 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:34.490634 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-rbac-proxy-cm podName:ba81a68e-6c72-4447-ad53-a81f29cee2a9 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:42.490619176 +0000 UTC m=+101.163153653 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-68vxh" (UID: "ba81a68e-6c72-4447-ad53-a81f29cee2a9") : configmap "kube-rbac-proxy" not found Jul 27 16:18:34.490946 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:34.490695 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba81a68e-6c72-4447-ad53-a81f29cee2a9-insights-runtime-extractor-tls podName:ba81a68e-6c72-4447-ad53-a81f29cee2a9 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:42.490677165 +0000 UTC m=+101.163211646 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ba81a68e-6c72-4447-ad53-a81f29cee2a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-68vxh" (UID: "ba81a68e-6c72-4447-ad53-a81f29cee2a9") : secret "insights-runtime-extractor-tls" not found Jul 27 16:18:37.007494 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:37.007451 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2337956f-94b4-44b9-9404-7cc9de6e6b0f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-j4h8x\" (UID: \"2337956f-94b4-44b9-9404-7cc9de6e6b0f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-j4h8x" Jul 27 16:18:37.007976 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:37.007566 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2528af38-f6e1-4c69-859a-6d00a5301ce4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-4skw2\" (UID: \"2528af38-f6e1-4c69-859a-6d00a5301ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2" Jul 27 16:18:37.010214 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:37.010181 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2337956f-94b4-44b9-9404-7cc9de6e6b0f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-j4h8x\" (UID: \"2337956f-94b4-44b9-9404-7cc9de6e6b0f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-j4h8x" Jul 27 16:18:37.010214 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:37.010193 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2528af38-f6e1-4c69-859a-6d00a5301ce4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-4skw2\" (UID: \"2528af38-f6e1-4c69-859a-6d00a5301ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2" Jul 27 16:18:37.108485 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:37.108449 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-kgxh2\" (UID: \"0e60c2da-59d2-4c42-bb33-3d4c22b3b90c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2" Jul 27 16:18:37.110927 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:37.110887 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e60c2da-59d2-4c42-bb33-3d4c22b3b90c-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-kgxh2\" (UID: \"0e60c2da-59d2-4c42-bb33-3d4c22b3b90c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2" Jul 27 16:18:37.137991 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:37.137966 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-j4h8x" Jul 27 16:18:37.143666 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:37.143650 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2" Jul 27 16:18:37.244402 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:37.244360 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2" Jul 27 16:18:37.291582 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:37.291529 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-j4m7l" podStartSLOduration=10.291509012 podStartE2EDuration="10.291509012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:29.24358831 +0000 UTC m=+87.916122810" watchObservedRunningTime="2026-07-27 16:18:37.291509012 +0000 UTC m=+95.964043511" Jul 27 16:18:37.292193 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:37.292176 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-j4h8x"] Jul 27 16:18:37.295018 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:18:37.294989 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2337956f_94b4_44b9_9404_7cc9de6e6b0f.slice/crio-da44928e3f400e7835507209db3ac0b85ed7574d90fa10aa8799cd5b4b4ea49a WatchSource:0}: Error finding container da44928e3f400e7835507209db3ac0b85ed7574d90fa10aa8799cd5b4b4ea49a: Status 404 returned error can't find the container with id da44928e3f400e7835507209db3ac0b85ed7574d90fa10aa8799cd5b4b4ea49a Jul 27 16:18:37.313208 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:37.313184 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2"] Jul 27 16:18:37.315529 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:18:37.315503 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2528af38_f6e1_4c69_859a_6d00a5301ce4.slice/crio-c0bcd42b5dcbd6575963d8c9f7018f5e35c05979211932038f4f09d8f2ebd513 WatchSource:0}: Error finding container c0bcd42b5dcbd6575963d8c9f7018f5e35c05979211932038f4f09d8f2ebd513: Status 404 returned error can't find the container with id c0bcd42b5dcbd6575963d8c9f7018f5e35c05979211932038f4f09d8f2ebd513 Jul 27 16:18:37.391310 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:37.391276 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2"] Jul 27 16:18:38.217866 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:38.217831 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2" event={"ID":"2528af38-f6e1-4c69-859a-6d00a5301ce4","Type":"ContainerStarted","Data":"c0bcd42b5dcbd6575963d8c9f7018f5e35c05979211932038f4f09d8f2ebd513"} Jul 27 16:18:38.219386 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:38.219356 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-j4h8x" event={"ID":"2337956f-94b4-44b9-9404-7cc9de6e6b0f","Type":"ContainerStarted","Data":"da44928e3f400e7835507209db3ac0b85ed7574d90fa10aa8799cd5b4b4ea49a"} Jul 27 16:18:38.220543 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:38.220513 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2" event={"ID":"0e60c2da-59d2-4c42-bb33-3d4c22b3b90c","Type":"ContainerStarted","Data":"05e4c8af92f92a20424f6acf5d4113b62cad71bf9cdec2fb80eeef1802b8fd70"} Jul 27 16:18:39.225924 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:39.225880 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-j4h8x" event={"ID":"2337956f-94b4-44b9-9404-7cc9de6e6b0f","Type":"ContainerStarted","Data":"60e672aca228a43ce0ae2b941ca67f4c18607ef4fe87dcc4d5d03dff6bc27ed1"} Jul 27 16:18:39.257440 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:39.257395 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-j4h8x" podStartSLOduration=17.029302094 podStartE2EDuration="18.257381395s" totalImagesPullingTime="1.228079301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:39.255853532 +0000 UTC m=+97.928388031" watchObservedRunningTime="2026-07-27 16:18:39.257381395 +0000 UTC m=+97.929915910" Jul 27 16:18:42.121248 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:42.121208 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-26l7z" Jul 27 16:18:42.546352 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:42.546312 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:42.546554 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:42.546394 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba81a68e-6c72-4447-ad53-a81f29cee2a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:42.547026 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:42.547005 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba81a68e-6c72-4447-ad53-a81f29cee2a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:42.549152 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:42.549127 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba81a68e-6c72-4447-ad53-a81f29cee2a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-68vxh\" (UID: \"ba81a68e-6c72-4447-ad53-a81f29cee2a9\") " pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:42.696780 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:42.696737 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-68vxh" Jul 27 16:18:43.331839 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:43.331814 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-68vxh"] Jul 27 16:18:43.334806 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:18:43.334778 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba81a68e_6c72_4447_ad53_a81f29cee2a9.slice/crio-0e1a1d34e1b56653987e5fe3acf885dd186e20693a36402b66c955cc71878152 WatchSource:0}: Error finding container 0e1a1d34e1b56653987e5fe3acf885dd186e20693a36402b66c955cc71878152: Status 404 returned error can't find the container with id 0e1a1d34e1b56653987e5fe3acf885dd186e20693a36402b66c955cc71878152 Jul 27 16:18:44.252546 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:44.252510 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2" event={"ID":"2528af38-f6e1-4c69-859a-6d00a5301ce4","Type":"ContainerStarted","Data":"9f2e25b638b40f449209d98fbf34694250cfe42289ed9447ee74bbbf10136be8"} Jul 27 16:18:44.253629 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:44.253599 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-68vxh" event={"ID":"ba81a68e-6c72-4447-ad53-a81f29cee2a9","Type":"ContainerStarted","Data":"0e1a1d34e1b56653987e5fe3acf885dd186e20693a36402b66c955cc71878152"} Jul 27 16:18:44.255166 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:44.255147 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2" event={"ID":"0e60c2da-59d2-4c42-bb33-3d4c22b3b90c","Type":"ContainerStarted","Data":"07efc919301678677a6b63dbf84c41fb162dabc09da44d205e3c707fe5dcbac5"} Jul 27 16:18:44.255254 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:44.255181 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2" event={"ID":"0e60c2da-59d2-4c42-bb33-3d4c22b3b90c","Type":"ContainerStarted","Data":"1911dbc5dd92fd0a3a3d6c23e07520b230023fb629076717530fd75114de707d"} Jul 27 16:18:44.282316 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:44.282271 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-4skw2" podStartSLOduration=17.431446509 podStartE2EDuration="23.282256689s" totalImagesPullingTime="5.85081018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:44.28158732 +0000 UTC m=+102.954121820" watchObservedRunningTime="2026-07-27 16:18:44.282256689 +0000 UTC m=+102.954791188" Jul 27 16:18:44.317705 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:44.317656 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kgxh2" podStartSLOduration=17.581170471 podStartE2EDuration="23.317643007s" totalImagesPullingTime="5.736472536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:44.314678794 +0000 UTC m=+102.987213293" watchObservedRunningTime="2026-07-27 16:18:44.317643007 +0000 UTC m=+102.990177571" Jul 27 16:18:45.170407 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:45.170371 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls\") pod \"dns-default-f7sqd\" (UID: \"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac\") " pod="openshift-dns/dns-default-f7sqd" Jul 27 16:18:45.172767 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:45.172747 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/76c82e0f-0e1b-400d-8c8f-9f5804aa15ac-metrics-tls\") pod \"dns-default-f7sqd\" (UID: \"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac\") " pod="openshift-dns/dns-default-f7sqd" Jul 27 16:18:45.447943 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:45.447845 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z2mnq\"" Jul 27 16:18:45.455138 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:45.455119 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f7sqd" Jul 27 16:18:45.609230 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:45.609197 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f7sqd"] Jul 27 16:18:45.613948 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:18:45.613917 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c82e0f_0e1b_400d_8c8f_9f5804aa15ac.slice/crio-91b85c2eabb13cf6ec785e26f574ac6c4ff15ff845cb90d2580fc22cdbcb3b83 WatchSource:0}: Error finding container 91b85c2eabb13cf6ec785e26f574ac6c4ff15ff845cb90d2580fc22cdbcb3b83: Status 404 returned error can't find the container with id 91b85c2eabb13cf6ec785e26f574ac6c4ff15ff845cb90d2580fc22cdbcb3b83 Jul 27 16:18:46.264158 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:46.264122 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f7sqd" event={"ID":"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac","Type":"ContainerStarted","Data":"91b85c2eabb13cf6ec785e26f574ac6c4ff15ff845cb90d2580fc22cdbcb3b83"} Jul 27 16:18:48.275762 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:48.275615 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f7sqd" event={"ID":"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac","Type":"ContainerStarted","Data":"be6a79a97fddb80777f37bcaddaa9e29dbd2e54825f3ba25d5d5bc71676e88b0"} Jul 27 16:18:48.275762 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:48.275675 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f7sqd" event={"ID":"76c82e0f-0e1b-400d-8c8f-9f5804aa15ac","Type":"ContainerStarted","Data":"43564f36b2dbf0de183480d00bbaaae995371ca879226a2dc59077e30c650ea1"} Jul 27 16:18:48.275762 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:48.275723 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-f7sqd" Jul 27 16:18:48.301528 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:48.301476 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-f7sqd" podStartSLOduration=65.627035956 podStartE2EDuration="1m7.301452684s" totalImagesPullingTime="1.674416728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:17:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:48.300121144 +0000 UTC m=+106.972655645" watchObservedRunningTime="2026-07-27 16:18:48.301452684 +0000 UTC m=+106.973987186" Jul 27 16:18:49.968313 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:49.968278 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8tbkw"] Jul 27 16:18:49.977886 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:49.977853 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8tbkw" Jul 27 16:18:49.983854 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:49.983830 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xbbpw\"" Jul 27 16:18:49.984073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:49.983830 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 27 16:18:50.002654 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:50.002626 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8tbkw"] Jul 27 16:18:50.116503 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:50.116468 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9b5e5814-4c80-42ab-88d8-593e29b22606-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8tbkw\" (UID: \"9b5e5814-4c80-42ab-88d8-593e29b22606\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8tbkw" Jul 27 16:18:50.217377 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:50.217342 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9b5e5814-4c80-42ab-88d8-593e29b22606-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8tbkw\" (UID: \"9b5e5814-4c80-42ab-88d8-593e29b22606\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8tbkw" Jul 27 16:18:50.220306 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:50.220231 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9b5e5814-4c80-42ab-88d8-593e29b22606-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8tbkw\" (UID: \"9b5e5814-4c80-42ab-88d8-593e29b22606\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8tbkw" Jul 27 16:18:50.288268 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:50.288235 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8tbkw" Jul 27 16:18:51.915258 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:51.915228 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8tbkw"] Jul 27 16:18:51.922789 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:18:51.922757 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5e5814_4c80_42ab_88d8_593e29b22606.slice/crio-b1181e9c0013ed5dc6f0ca5afe1c236abc14f72f9709d520c748ba751f38f34a WatchSource:0}: Error finding container b1181e9c0013ed5dc6f0ca5afe1c236abc14f72f9709d520c748ba751f38f34a: Status 404 returned error can't find the container with id b1181e9c0013ed5dc6f0ca5afe1c236abc14f72f9709d520c748ba751f38f34a Jul 27 16:18:52.294809 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:52.294772 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8tbkw" event={"ID":"9b5e5814-4c80-42ab-88d8-593e29b22606","Type":"ContainerStarted","Data":"b1181e9c0013ed5dc6f0ca5afe1c236abc14f72f9709d520c748ba751f38f34a"} Jul 27 16:18:52.296381 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:52.296343 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-68vxh" event={"ID":"ba81a68e-6c72-4447-ad53-a81f29cee2a9","Type":"ContainerStarted","Data":"95a7b55447a2fa682c8a485453e0412b3364ecf9bb38d3988d3914dae66844f7"} Jul 27 16:18:53.301812 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:53.301771 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8tbkw" event={"ID":"9b5e5814-4c80-42ab-88d8-593e29b22606","Type":"ContainerStarted","Data":"a7f0a27a6ad2b9f50bbe0e1afb26ddd0d45d8ccf1e6d3eede4ffcaf7c50cda9d"} Jul 27 16:18:53.302287 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:53.302017 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8tbkw" Jul 27 16:18:53.307298 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:53.307276 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8tbkw" Jul 27 16:18:53.340114 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:53.340007 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8tbkw" podStartSLOduration=3.184503566 podStartE2EDuration="4.339988561s" totalImagesPullingTime="1.155484995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:53.338290173 +0000 UTC m=+112.010824675" watchObservedRunningTime="2026-07-27 16:18:53.339988561 +0000 UTC m=+112.012523059" Jul 27 16:18:54.350644 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:54.350612 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-gcx88"] Jul 27 16:18:54.367575 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:54.367550 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-gcx88" Jul 27 16:18:54.370604 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:54.369122 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-gcx88"] Jul 27 16:18:54.372631 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:54.372074 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 27 16:18:54.372631 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:54.372339 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 27 16:18:54.372631 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:54.372476 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 27 16:18:54.372631 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:54.372536 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-c6wnv\"" Jul 27 16:18:54.459189 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:54.459154 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dba214b-8b12-4185-af7b-c5cc09216918-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-gcx88\" (UID: \"5dba214b-8b12-4185-af7b-c5cc09216918\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-gcx88" Jul 27 16:18:54.459382 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:54.459213 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5dba214b-8b12-4185-af7b-c5cc09216918-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-gcx88\" (UID: \"5dba214b-8b12-4185-af7b-c5cc09216918\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-gcx88" Jul 27 16:18:54.459382 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:54.459239 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px596\" (UniqueName: \"kubernetes.io/projected/5dba214b-8b12-4185-af7b-c5cc09216918-kube-api-access-px596\") pod \"prometheus-operator-84d69f794d-gcx88\" (UID: \"5dba214b-8b12-4185-af7b-c5cc09216918\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-gcx88" Jul 27 16:18:54.459490 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:54.459392 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5dba214b-8b12-4185-af7b-c5cc09216918-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-gcx88\" (UID: \"5dba214b-8b12-4185-af7b-c5cc09216918\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-gcx88" Jul 27 16:18:54.560752 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:54.560713 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5dba214b-8b12-4185-af7b-c5cc09216918-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-gcx88\" (UID: \"5dba214b-8b12-4185-af7b-c5cc09216918\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-gcx88" Jul 27 16:18:54.560917 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:54.560769 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dba214b-8b12-4185-af7b-c5cc09216918-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-gcx88\" (UID: \"5dba214b-8b12-4185-af7b-c5cc09216918\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-gcx88" Jul 27 16:18:54.560917 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:54.560817 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5dba214b-8b12-4185-af7b-c5cc09216918-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-gcx88\" (UID: \"5dba214b-8b12-4185-af7b-c5cc09216918\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-gcx88" Jul 27 16:18:54.560917 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:54.560844 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px596\" (UniqueName: \"kubernetes.io/projected/5dba214b-8b12-4185-af7b-c5cc09216918-kube-api-access-px596\") pod \"prometheus-operator-84d69f794d-gcx88\" (UID: \"5dba214b-8b12-4185-af7b-c5cc09216918\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-gcx88" Jul 27 16:18:54.561023 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:54.560943 2550 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 27 16:18:54.561023 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:18:54.561004 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dba214b-8b12-4185-af7b-c5cc09216918-prometheus-operator-tls podName:5dba214b-8b12-4185-af7b-c5cc09216918 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:55.060987549 +0000 UTC m=+113.733522046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/5dba214b-8b12-4185-af7b-c5cc09216918-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-gcx88" (UID: "5dba214b-8b12-4185-af7b-c5cc09216918") : secret "prometheus-operator-tls" not found Jul 27 16:18:54.561499 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:54.561483 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5dba214b-8b12-4185-af7b-c5cc09216918-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-gcx88\" (UID: \"5dba214b-8b12-4185-af7b-c5cc09216918\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-gcx88" Jul 27 16:18:54.563412 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:54.563386 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5dba214b-8b12-4185-af7b-c5cc09216918-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-gcx88\" (UID: \"5dba214b-8b12-4185-af7b-c5cc09216918\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-gcx88" Jul 27 16:18:54.573248 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:54.573224 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px596\" (UniqueName: \"kubernetes.io/projected/5dba214b-8b12-4185-af7b-c5cc09216918-kube-api-access-px596\") pod \"prometheus-operator-84d69f794d-gcx88\" (UID: \"5dba214b-8b12-4185-af7b-c5cc09216918\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-gcx88" Jul 27 16:18:55.065122 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:55.065086 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dba214b-8b12-4185-af7b-c5cc09216918-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-gcx88\" (UID: \"5dba214b-8b12-4185-af7b-c5cc09216918\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-gcx88" Jul 27 16:18:55.068028 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:55.067996 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dba214b-8b12-4185-af7b-c5cc09216918-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-gcx88\" (UID: \"5dba214b-8b12-4185-af7b-c5cc09216918\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-gcx88" Jul 27 16:18:55.279530 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:55.279513 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-gcx88" Jul 27 16:18:55.422686 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:55.422652 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-gcx88"] Jul 27 16:18:55.425687 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:18:55.425653 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dba214b_8b12_4185_af7b_c5cc09216918.slice/crio-6e774bc54fd406af5e10d429c946470e6ac437f1c215fef6197c08ad19835e22 WatchSource:0}: Error finding container 6e774bc54fd406af5e10d429c946470e6ac437f1c215fef6197c08ad19835e22: Status 404 returned error can't find the container with id 6e774bc54fd406af5e10d429c946470e6ac437f1c215fef6197c08ad19835e22 Jul 27 16:18:56.314749 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:56.314715 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-68vxh" event={"ID":"ba81a68e-6c72-4447-ad53-a81f29cee2a9","Type":"ContainerStarted","Data":"926576f08c46b070079f05a0cf956d1e9f2991acefbe7799e99e88b4124e7e15"} Jul 27 16:18:56.315883 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:56.315858 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-gcx88" event={"ID":"5dba214b-8b12-4185-af7b-c5cc09216918","Type":"ContainerStarted","Data":"6e774bc54fd406af5e10d429c946470e6ac437f1c215fef6197c08ad19835e22"} Jul 27 16:18:58.282962 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:18:58.282934 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-f7sqd" Jul 27 16:19:00.331784 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:00.331753 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-68vxh" event={"ID":"ba81a68e-6c72-4447-ad53-a81f29cee2a9","Type":"ContainerStarted","Data":"ee464a7a4db013a0a792b8d67d9d6604782589c8905190dd6d1130ff55b35364"} Jul 27 16:19:01.336635 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:01.336588 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-gcx88" event={"ID":"5dba214b-8b12-4185-af7b-c5cc09216918","Type":"ContainerStarted","Data":"d6cf2286100618addba88d940b9bb36ecc28d1c6570ecd55ab2690e04ed4f0e1"} Jul 27 16:19:01.337128 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:01.336679 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-gcx88" event={"ID":"5dba214b-8b12-4185-af7b-c5cc09216918","Type":"ContainerStarted","Data":"4f9e20acd63845c44ef57f04fe22bd3e2065cd568da0d499e270524f797e5337"} Jul 27 16:19:01.357949 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:01.357884 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-68vxh" podStartSLOduration=19.418339905 podStartE2EDuration="35.357868701s" totalImagesPullingTime="15.939528796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:18:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:00.360803575 +0000 UTC m=+119.033338071" watchObservedRunningTime="2026-07-27 16:19:01.357868701 +0000 UTC m=+120.030403199" Jul 27 16:19:01.358324 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:01.358295 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-gcx88" podStartSLOduration=2.491674155 podStartE2EDuration="7.358287066s" totalImagesPullingTime="4.866612911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:01.357726291 +0000 UTC m=+120.030260791" watchObservedRunningTime="2026-07-27 16:19:01.358287066 +0000 UTC m=+120.030821565" Jul 27 16:19:03.773786 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:03.773753 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vndgf"] Jul 27 16:19:03.776778 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:03.776759 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:03.779572 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:03.779547 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 27 16:19:03.779680 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:03.779652 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 27 16:19:03.779776 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:03.779755 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 27 16:19:03.780064 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:03.780049 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wzslv\"" Jul 27 16:19:03.940045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:03.940013 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:03.940226 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:03.940053 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-node-exporter-tls\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:03.940226 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:03.940150 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-sys\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:03.940226 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:03.940179 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-metrics-client-ca\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:03.940226 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:03.940201 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8crh\" (UniqueName: \"kubernetes.io/projected/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-kube-api-access-n8crh\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:03.940387 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:03.940247 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-node-exporter-wtmp\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:03.940387 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:03.940288 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-node-exporter-textfile\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:03.940387 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:03.940331 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-root\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:03.940387 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:03.940357 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-node-exporter-accelerators-collector-config\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.041524 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.041441 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-node-exporter-textfile\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.041524 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.041479 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-root\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.041524 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.041495 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-node-exporter-accelerators-collector-config\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.041524 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.041515 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.041933 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.041539 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-node-exporter-tls\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.041933 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.041591 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-root\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.041933 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.041601 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-sys\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.041933 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.041641 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-sys\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.041933 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.041659 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-metrics-client-ca\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.041933 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.041750 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8crh\" (UniqueName: \"kubernetes.io/projected/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-kube-api-access-n8crh\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.041933 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.041813 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-node-exporter-wtmp\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.042282 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.041933 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-node-exporter-textfile\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.042282 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.042071 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-node-exporter-wtmp\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.042282 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.042239 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-node-exporter-accelerators-collector-config\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.042282 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.042265 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-metrics-client-ca\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.044080 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.044060 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-node-exporter-tls\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.044169 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.044113 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.051425 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.051399 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8crh\" (UniqueName: \"kubernetes.io/projected/60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090-kube-api-access-n8crh\") pod \"node-exporter-vndgf\" (UID: \"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090\") " pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.086489 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.086459 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vndgf" Jul 27 16:19:04.097441 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:19:04.097414 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b9b3b2_5ab3_4bcc_a17a_ff64fcbe4090.slice/crio-dc599529325d7b1a9af8eabd008e9e7465820ead4ab5c3df52033f373e6bf720 WatchSource:0}: Error finding container dc599529325d7b1a9af8eabd008e9e7465820ead4ab5c3df52033f373e6bf720: Status 404 returned error can't find the container with id dc599529325d7b1a9af8eabd008e9e7465820ead4ab5c3df52033f373e6bf720 Jul 27 16:19:04.348374 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.348288 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vndgf" event={"ID":"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090","Type":"ContainerStarted","Data":"dc599529325d7b1a9af8eabd008e9e7465820ead4ab5c3df52033f373e6bf720"} Jul 27 16:19:04.813771 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.813737 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 16:19:04.818015 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.817992 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:04.821180 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.821155 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 27 16:19:04.821316 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.821213 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 27 16:19:04.821530 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.821495 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 27 16:19:04.821620 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.821538 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 27 16:19:04.821620 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.821502 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 27 16:19:04.821753 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.821737 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ln7ps\"" Jul 27 16:19:04.821834 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.821817 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 27 16:19:04.821961 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.821944 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 27 16:19:04.822044 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.821981 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 27 16:19:04.822243 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.822229 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 27 16:19:04.842474 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.842442 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 16:19:04.949626 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.949592 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btt2p\" (UniqueName: \"kubernetes.io/projected/bf37d2a1-2ccb-49e7-9a33-80b61390d132-kube-api-access-btt2p\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:04.949626 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.949629 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:04.949844 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.949650 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:04.949844 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.949755 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bf37d2a1-2ccb-49e7-9a33-80b61390d132-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:04.949844 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.949790 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf37d2a1-2ccb-49e7-9a33-80b61390d132-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:04.949844 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.949820 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:04.950014 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.949848 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf37d2a1-2ccb-49e7-9a33-80b61390d132-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:04.950014 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.949875 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-config-volume\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:04.950014 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.949951 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf37d2a1-2ccb-49e7-9a33-80b61390d132-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:04.950014 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.949989 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-web-config\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:04.950014 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.950006 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:04.950178 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.950031 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:04.950178 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:04.950050 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf37d2a1-2ccb-49e7-9a33-80b61390d132-config-out\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.050914 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.050863 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.050914 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.050915 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf37d2a1-2ccb-49e7-9a33-80b61390d132-config-out\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.051137 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.050948 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btt2p\" (UniqueName: \"kubernetes.io/projected/bf37d2a1-2ccb-49e7-9a33-80b61390d132-kube-api-access-btt2p\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.051137 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.050975 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.051137 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.050991 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.051137 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.051021 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bf37d2a1-2ccb-49e7-9a33-80b61390d132-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.051137 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.051036 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf37d2a1-2ccb-49e7-9a33-80b61390d132-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.051393 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.051365 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.051466 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.051416 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf37d2a1-2ccb-49e7-9a33-80b61390d132-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.051466 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.051450 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-config-volume\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.051565 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.051476 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bf37d2a1-2ccb-49e7-9a33-80b61390d132-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.051565 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.051490 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf37d2a1-2ccb-49e7-9a33-80b61390d132-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.051825 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.051601 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-web-config\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.051825 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.051638 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.052070 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.052017 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf37d2a1-2ccb-49e7-9a33-80b61390d132-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.053422 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.052774 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf37d2a1-2ccb-49e7-9a33-80b61390d132-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.054226 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.054194 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf37d2a1-2ccb-49e7-9a33-80b61390d132-config-out\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.055868 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.055840 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.056362 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.056337 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-web-config\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.056535 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.056510 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.056653 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.056634 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf37d2a1-2ccb-49e7-9a33-80b61390d132-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.056931 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.056884 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.057109 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.057083 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.057200 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.057180 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-config-volume\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.062723 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.062701 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.070592 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.070540 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btt2p\" (UniqueName: \"kubernetes.io/projected/bf37d2a1-2ccb-49e7-9a33-80b61390d132-kube-api-access-btt2p\") pod \"alertmanager-main-0\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.128238 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.128206 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:19:05.285825 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.285719 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 16:19:05.288296 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:19:05.288264 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf37d2a1_2ccb_49e7_9a33_80b61390d132.slice/crio-aa7a5e9835d250bd9479c5837f102fc94686ee9562beef3f92e9506f9ad033bd WatchSource:0}: Error finding container aa7a5e9835d250bd9479c5837f102fc94686ee9562beef3f92e9506f9ad033bd: Status 404 returned error can't find the container with id aa7a5e9835d250bd9479c5837f102fc94686ee9562beef3f92e9506f9ad033bd Jul 27 16:19:05.353728 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:05.353651 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf37d2a1-2ccb-49e7-9a33-80b61390d132","Type":"ContainerStarted","Data":"aa7a5e9835d250bd9479c5837f102fc94686ee9562beef3f92e9506f9ad033bd"} Jul 27 16:19:06.711523 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.711488 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6456df978d-5vg66"] Jul 27 16:19:06.714784 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.714767 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.717657 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.717625 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 27 16:19:06.717778 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.717761 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 27 16:19:06.717863 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.717847 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4dzpd\"" Jul 27 16:19:06.717984 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.717970 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6a1qrga8erffs\"" Jul 27 16:19:06.718037 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.718008 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 27 16:19:06.718148 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.718130 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 27 16:19:06.718837 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.718822 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 27 16:19:06.730892 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.730871 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6456df978d-5vg66"] Jul 27 16:19:06.766519 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.766485 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3e3b341d-cd31-447e-b15c-04289beef65b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.766690 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.766543 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e3b341d-cd31-447e-b15c-04289beef65b-metrics-client-ca\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.766690 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.766574 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e3b341d-cd31-447e-b15c-04289beef65b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.766690 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.766610 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3e3b341d-cd31-447e-b15c-04289beef65b-secret-thanos-querier-tls\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.766690 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.766647 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3e3b341d-cd31-447e-b15c-04289beef65b-secret-grpc-tls\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.766690 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.766669 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f79l\" (UniqueName: \"kubernetes.io/projected/3e3b341d-cd31-447e-b15c-04289beef65b-kube-api-access-6f79l\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.766950 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.766722 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3e3b341d-cd31-447e-b15c-04289beef65b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.766950 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.766756 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e3b341d-cd31-447e-b15c-04289beef65b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.867457 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.867412 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3e3b341d-cd31-447e-b15c-04289beef65b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.867630 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.867498 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e3b341d-cd31-447e-b15c-04289beef65b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.867630 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.867548 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3e3b341d-cd31-447e-b15c-04289beef65b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.867630 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.867591 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e3b341d-cd31-447e-b15c-04289beef65b-metrics-client-ca\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.867630 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.867620 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e3b341d-cd31-447e-b15c-04289beef65b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.867841 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.867653 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3e3b341d-cd31-447e-b15c-04289beef65b-secret-thanos-querier-tls\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.867841 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.867691 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3e3b341d-cd31-447e-b15c-04289beef65b-secret-grpc-tls\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.867841 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.867714 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f79l\" (UniqueName: \"kubernetes.io/projected/3e3b341d-cd31-447e-b15c-04289beef65b-kube-api-access-6f79l\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.868608 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.868558 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e3b341d-cd31-447e-b15c-04289beef65b-metrics-client-ca\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.871489 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.871349 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e3b341d-cd31-447e-b15c-04289beef65b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.871489 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.871404 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e3b341d-cd31-447e-b15c-04289beef65b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.871799 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.871779 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3e3b341d-cd31-447e-b15c-04289beef65b-secret-thanos-querier-tls\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.871874 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.871831 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3e3b341d-cd31-447e-b15c-04289beef65b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.871874 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.871833 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3e3b341d-cd31-447e-b15c-04289beef65b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.871874 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.871865 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3e3b341d-cd31-447e-b15c-04289beef65b-secret-grpc-tls\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:06.878219 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:06.878187 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f79l\" (UniqueName: \"kubernetes.io/projected/3e3b341d-cd31-447e-b15c-04289beef65b-kube-api-access-6f79l\") pod \"thanos-querier-6456df978d-5vg66\" (UID: \"3e3b341d-cd31-447e-b15c-04289beef65b\") " pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:07.027256 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:07.027205 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:07.409056 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:07.409025 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6456df978d-5vg66"] Jul 27 16:19:07.446825 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:19:07.446792 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3b341d_cd31_447e_b15c_04289beef65b.slice/crio-c4c426b2011b6d92e27211baeabcd9c99692530592535514f855365831bcb6ca WatchSource:0}: Error finding container c4c426b2011b6d92e27211baeabcd9c99692530592535514f855365831bcb6ca: Status 404 returned error can't find the container with id c4c426b2011b6d92e27211baeabcd9c99692530592535514f855365831bcb6ca Jul 27 16:19:07.629360 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:07.629280 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-l48kt"] Jul 27 16:19:07.632408 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:07.632377 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-l48kt" Jul 27 16:19:07.636064 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:07.636039 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-zwrx4\"" Jul 27 16:19:07.636196 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:07.636085 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 27 16:19:07.636196 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:07.636041 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 27 16:19:07.646081 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:07.646058 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-l48kt"] Jul 27 16:19:07.674602 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:07.674571 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpchq\" (UniqueName: \"kubernetes.io/projected/223f7f5e-f930-4bfc-958d-9ec54d0a27a9-kube-api-access-tpchq\") pod \"downloads-7c97dc4555-l48kt\" (UID: \"223f7f5e-f930-4bfc-958d-9ec54d0a27a9\") " pod="openshift-console/downloads-7c97dc4555-l48kt" Jul 27 16:19:07.775720 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:07.775687 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpchq\" (UniqueName: \"kubernetes.io/projected/223f7f5e-f930-4bfc-958d-9ec54d0a27a9-kube-api-access-tpchq\") pod \"downloads-7c97dc4555-l48kt\" (UID: \"223f7f5e-f930-4bfc-958d-9ec54d0a27a9\") " pod="openshift-console/downloads-7c97dc4555-l48kt" Jul 27 16:19:07.785151 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:07.785119 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpchq\" (UniqueName: \"kubernetes.io/projected/223f7f5e-f930-4bfc-958d-9ec54d0a27a9-kube-api-access-tpchq\") pod \"downloads-7c97dc4555-l48kt\" (UID: \"223f7f5e-f930-4bfc-958d-9ec54d0a27a9\") " pod="openshift-console/downloads-7c97dc4555-l48kt" Jul 27 16:19:07.944072 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:07.943979 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-l48kt" Jul 27 16:19:08.101223 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.101189 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-l48kt"] Jul 27 16:19:08.104238 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:19:08.104202 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223f7f5e_f930_4bfc_958d_9ec54d0a27a9.slice/crio-79a20376e66acef72ce75e42682852f51aafc1791ce14ab3372448d2e76f514b WatchSource:0}: Error finding container 79a20376e66acef72ce75e42682852f51aafc1791ce14ab3372448d2e76f514b: Status 404 returned error can't find the container with id 79a20376e66acef72ce75e42682852f51aafc1791ce14ab3372448d2e76f514b Jul 27 16:19:08.107047 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.107023 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l"] Jul 27 16:19:08.109712 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.109688 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.113873 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.113851 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 27 16:19:08.114419 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.114402 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 27 16:19:08.115634 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.115610 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 27 16:19:08.115725 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.115693 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7vcu1sl8tc2m5\"" Jul 27 16:19:08.115845 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.115821 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-sb78g\"" Jul 27 16:19:08.115845 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.115831 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 27 16:19:08.124760 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.124737 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l"] Jul 27 16:19:08.178489 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.178463 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-secret-metrics-server-client-certs\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.178647 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.178498 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.178647 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.178528 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-client-ca-bundle\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.178647 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.178581 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-metrics-server-audit-profiles\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.178777 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.178670 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-audit-log\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.178777 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.178705 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flgnc\" (UniqueName: \"kubernetes.io/projected/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-kube-api-access-flgnc\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.178777 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.178749 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-secret-metrics-server-tls\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.279098 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.279058 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-secret-metrics-server-client-certs\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.279098 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.279097 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.279303 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.279116 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-client-ca-bundle\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.279303 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.279132 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-metrics-server-audit-profiles\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.279303 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.279176 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-audit-log\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.279456 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.279306 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flgnc\" (UniqueName: \"kubernetes.io/projected/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-kube-api-access-flgnc\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.279456 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.279366 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-secret-metrics-server-tls\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.279621 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.279571 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-audit-log\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.280075 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.280051 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.280543 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.280521 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-metrics-server-audit-profiles\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.282017 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.281988 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-client-ca-bundle\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.282106 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.282040 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-secret-metrics-server-tls\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.282150 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.282126 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-secret-metrics-server-client-certs\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.290397 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.290374 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flgnc\" (UniqueName: \"kubernetes.io/projected/6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9-kube-api-access-flgnc\") pod \"metrics-server-76d9d7c7f8-5rn2l\" (UID: \"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9\") " pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.367620 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.367581 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-l48kt" event={"ID":"223f7f5e-f930-4bfc-958d-9ec54d0a27a9","Type":"ContainerStarted","Data":"79a20376e66acef72ce75e42682852f51aafc1791ce14ab3372448d2e76f514b"} Jul 27 16:19:08.369645 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.369618 2550 generic.go:356] "Generic (PLEG): container finished" podID="60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090" containerID="d5839a1ae44ab347058297e34ca3f3c595c68ad4ccf7be074454c5951eb3b375" exitCode=0 Jul 27 16:19:08.369780 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.369691 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vndgf" event={"ID":"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090","Type":"ContainerDied","Data":"d5839a1ae44ab347058297e34ca3f3c595c68ad4ccf7be074454c5951eb3b375"} Jul 27 16:19:08.371217 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.371188 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" event={"ID":"3e3b341d-cd31-447e-b15c-04289beef65b","Type":"ContainerStarted","Data":"c4c426b2011b6d92e27211baeabcd9c99692530592535514f855365831bcb6ca"} Jul 27 16:19:08.372889 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.372863 2550 generic.go:356] "Generic (PLEG): container finished" podID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerID="a7f880875ba319f4bd5d82b5ff634aaf91dbd4cc5b6565e3a493bc82f14da5be" exitCode=0 Jul 27 16:19:08.373069 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.372945 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf37d2a1-2ccb-49e7-9a33-80b61390d132","Type":"ContainerDied","Data":"a7f880875ba319f4bd5d82b5ff634aaf91dbd4cc5b6565e3a493bc82f14da5be"} Jul 27 16:19:08.419593 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.419564 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:08.492780 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.492238 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-x9964"] Jul 27 16:19:08.496099 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.495492 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x9964" Jul 27 16:19:08.501275 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.501251 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 27 16:19:08.502063 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.502040 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wv96t\"" Jul 27 16:19:08.510623 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.510590 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-x9964"] Jul 27 16:19:08.583022 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.582972 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a364884-ca85-47df-9f19-b2789c22c312-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-x9964\" (UID: \"6a364884-ca85-47df-9f19-b2789c22c312\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x9964" Jul 27 16:19:08.583206 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.583071 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88cc5\" (UniqueName: \"kubernetes.io/projected/6a364884-ca85-47df-9f19-b2789c22c312-kube-api-access-88cc5\") pod \"monitoring-plugin-68b7fc9574-x9964\" (UID: \"6a364884-ca85-47df-9f19-b2789c22c312\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x9964" Jul 27 16:19:08.619848 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.619820 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l"] Jul 27 16:19:08.622461 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:19:08.622433 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c20cd6a_6b94_4fe7_8bf7_ee7b00fd60c9.slice/crio-935935d7bca6f4c2fcc5db73a8e1d46d86c253b8805f1c252c0b40b2f079b291 WatchSource:0}: Error finding container 935935d7bca6f4c2fcc5db73a8e1d46d86c253b8805f1c252c0b40b2f079b291: Status 404 returned error can't find the container with id 935935d7bca6f4c2fcc5db73a8e1d46d86c253b8805f1c252c0b40b2f079b291 Jul 27 16:19:08.683745 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.683717 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a364884-ca85-47df-9f19-b2789c22c312-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-x9964\" (UID: \"6a364884-ca85-47df-9f19-b2789c22c312\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x9964" Jul 27 16:19:08.683946 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.683792 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88cc5\" (UniqueName: \"kubernetes.io/projected/6a364884-ca85-47df-9f19-b2789c22c312-kube-api-access-88cc5\") pod \"monitoring-plugin-68b7fc9574-x9964\" (UID: \"6a364884-ca85-47df-9f19-b2789c22c312\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x9964" Jul 27 16:19:08.683946 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:19:08.683888 2550 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 27 16:19:08.684068 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:19:08.683973 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a364884-ca85-47df-9f19-b2789c22c312-monitoring-plugin-cert podName:6a364884-ca85-47df-9f19-b2789c22c312 nodeName:}" failed. No retries permitted until 2026-07-27 16:19:09.183958235 +0000 UTC m=+127.856492713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/6a364884-ca85-47df-9f19-b2789c22c312-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-x9964" (UID: "6a364884-ca85-47df-9f19-b2789c22c312") : secret "monitoring-plugin-cert" not found Jul 27 16:19:08.694711 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:08.694683 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88cc5\" (UniqueName: \"kubernetes.io/projected/6a364884-ca85-47df-9f19-b2789c22c312-kube-api-access-88cc5\") pod \"monitoring-plugin-68b7fc9574-x9964\" (UID: \"6a364884-ca85-47df-9f19-b2789c22c312\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x9964" Jul 27 16:19:09.188211 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:09.188173 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a364884-ca85-47df-9f19-b2789c22c312-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-x9964\" (UID: \"6a364884-ca85-47df-9f19-b2789c22c312\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x9964" Jul 27 16:19:09.191474 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:09.191450 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a364884-ca85-47df-9f19-b2789c22c312-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-x9964\" (UID: \"6a364884-ca85-47df-9f19-b2789c22c312\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x9964" Jul 27 16:19:09.382062 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:09.382024 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vndgf" event={"ID":"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090","Type":"ContainerStarted","Data":"bd582e96dd1ba5006715bb6ad5437aafe2ff330680d657b5c50e925aa1d3b7cc"} Jul 27 16:19:09.382234 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:09.382097 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vndgf" event={"ID":"60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090","Type":"ContainerStarted","Data":"e9a68a6203193e58065efdab3225cebc6ad4aa219232f1cf88d84d33a19794b9"} Jul 27 16:19:09.384587 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:09.383702 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" event={"ID":"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9","Type":"ContainerStarted","Data":"935935d7bca6f4c2fcc5db73a8e1d46d86c253b8805f1c252c0b40b2f079b291"} Jul 27 16:19:09.422006 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:09.421971 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x9964" Jul 27 16:19:10.974423 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:10.974326 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vndgf" podStartSLOduration=4.010473254 podStartE2EDuration="7.974281931s" totalImagesPullingTime="3.887024191s" totalInitContainerRuntime="76.784486ms" isStatefulPod=true podCreationTimestamp="2026-07-27 16:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:09.408975867 +0000 UTC m=+128.081510367" watchObservedRunningTime="2026-07-27 16:19:10.974281931 +0000 UTC m=+129.646816410" Jul 27 16:19:10.976234 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:10.976206 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-x9964"] Jul 27 16:19:10.981250 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:19:10.981227 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a364884_ca85_47df_9f19_b2789c22c312.slice/crio-de686962f65d017d85ba8b9a8461f2603306a1f766ca8ccdf30213da6bcf2fc5 WatchSource:0}: Error finding container de686962f65d017d85ba8b9a8461f2603306a1f766ca8ccdf30213da6bcf2fc5: Status 404 returned error can't find the container with id de686962f65d017d85ba8b9a8461f2603306a1f766ca8ccdf30213da6bcf2fc5 Jul 27 16:19:11.394846 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:11.394747 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x9964" event={"ID":"6a364884-ca85-47df-9f19-b2789c22c312","Type":"ContainerStarted","Data":"de686962f65d017d85ba8b9a8461f2603306a1f766ca8ccdf30213da6bcf2fc5"} Jul 27 16:19:11.396844 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:11.396819 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" event={"ID":"3e3b341d-cd31-447e-b15c-04289beef65b","Type":"ContainerStarted","Data":"79bc09d7e97fdb2dd0f4c08bcf4a334b9ca5c90c119614bfc3b18e56229c910a"} Jul 27 16:19:11.396987 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:11.396874 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" event={"ID":"3e3b341d-cd31-447e-b15c-04289beef65b","Type":"ContainerStarted","Data":"098f62ef92d85a8f2d179b32215a6756977b5f6fcd7e995304609c2836c0f3c1"} Jul 27 16:19:11.396987 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:11.396925 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" event={"ID":"3e3b341d-cd31-447e-b15c-04289beef65b","Type":"ContainerStarted","Data":"ecfcad560ee399e6d19f60e2032d24089712baf4b735a00716f52149ddf31998"} Jul 27 16:19:11.399559 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:11.399541 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf37d2a1-2ccb-49e7-9a33-80b61390d132","Type":"ContainerStarted","Data":"e8a63c095ad34c28d564870d8ebdc6b41f80363c43d6effbf489a0804ce9093b"} Jul 27 16:19:11.399651 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:11.399635 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf37d2a1-2ccb-49e7-9a33-80b61390d132","Type":"ContainerStarted","Data":"1cb22c6beda5be30861513e79b0baed7ea35efd4ed737f4d0ffd6a985eedb79e"} Jul 27 16:19:11.399701 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:11.399684 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf37d2a1-2ccb-49e7-9a33-80b61390d132","Type":"ContainerStarted","Data":"b0a76cedbd833631d03b2af7b54cd16cce8b8ce309c32b7bab27f8e0c2a5354c"} Jul 27 16:19:11.399732 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:11.399725 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf37d2a1-2ccb-49e7-9a33-80b61390d132","Type":"ContainerStarted","Data":"d68e31e472598c7264a395cd75095e7d1781d94b863463c67071537eac60dee7"} Jul 27 16:19:11.399772 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:11.399764 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf37d2a1-2ccb-49e7-9a33-80b61390d132","Type":"ContainerStarted","Data":"f56ac2d66adab968a35ba13cf947f1a6a590ce145b855ca6b1bc6260dd6e6d42"} Jul 27 16:19:11.400814 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:11.400792 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" event={"ID":"6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9","Type":"ContainerStarted","Data":"e866efa55c97ec27915007c07b5e5c9ee531e49dbad5df08eab8638ea735a213"} Jul 27 16:19:11.426646 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:11.426598 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" podStartSLOduration=1.268242705 podStartE2EDuration="3.426583138s" totalImagesPullingTime="2.158340433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:11.425737222 +0000 UTC m=+130.098271722" watchObservedRunningTime="2026-07-27 16:19:11.426583138 +0000 UTC m=+130.099117615" Jul 27 16:19:11.523265 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:11.523233 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs\") pod \"network-metrics-daemon-mg2dr\" (UID: \"fc3e4156-6cfc-430a-8842-a9d801cfb9f0\") " pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:19:11.525782 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:11.525762 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3e4156-6cfc-430a-8842-a9d801cfb9f0-metrics-certs\") pod \"network-metrics-daemon-mg2dr\" (UID: \"fc3e4156-6cfc-430a-8842-a9d801cfb9f0\") " pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:19:11.562661 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:11.562631 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9d9gv\"" Jul 27 16:19:11.569897 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:11.569866 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg2dr" Jul 27 16:19:11.726368 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:11.726298 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mg2dr"] Jul 27 16:19:11.827643 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:19:11.827608 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3e4156_6cfc_430a_8842_a9d801cfb9f0.slice/crio-69eb94cfb748e1221b99fbf9273063de0dae52bc6fbb45b2a886fea15492a5cd WatchSource:0}: Error finding container 69eb94cfb748e1221b99fbf9273063de0dae52bc6fbb45b2a886fea15492a5cd: Status 404 returned error can't find the container with id 69eb94cfb748e1221b99fbf9273063de0dae52bc6fbb45b2a886fea15492a5cd Jul 27 16:19:12.409448 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:12.409409 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" event={"ID":"3e3b341d-cd31-447e-b15c-04289beef65b","Type":"ContainerStarted","Data":"9508d2ca5d27bb2ce174e23924992350a6a4edaeb0497680490d2a0398fa124b"} Jul 27 16:19:12.409997 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:12.409545 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:12.409997 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:12.409585 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" event={"ID":"3e3b341d-cd31-447e-b15c-04289beef65b","Type":"ContainerStarted","Data":"30213dd951dba7d68ec5bc54b9cb83383727c5cf1b6de11054ac089e086a071b"} Jul 27 16:19:12.409997 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:12.409624 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" event={"ID":"3e3b341d-cd31-447e-b15c-04289beef65b","Type":"ContainerStarted","Data":"49636ce97235428de13c1b51e7bed493392e02e0f7e4a23332d5e3e57b6e1809"} Jul 27 16:19:12.412648 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:12.412622 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf37d2a1-2ccb-49e7-9a33-80b61390d132","Type":"ContainerStarted","Data":"94ed3c68bae59e97ef7049a2c3fc81da5d97720db15ef867d2a63e062ba98100"} Jul 27 16:19:12.413836 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:12.413810 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mg2dr" event={"ID":"fc3e4156-6cfc-430a-8842-a9d801cfb9f0","Type":"ContainerStarted","Data":"69eb94cfb748e1221b99fbf9273063de0dae52bc6fbb45b2a886fea15492a5cd"} Jul 27 16:19:12.443652 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:12.443605 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" podStartSLOduration=2.276595041 podStartE2EDuration="6.443588922s" totalImagesPullingTime="4.166993881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:19:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:12.442659925 +0000 UTC m=+131.115194425" watchObservedRunningTime="2026-07-27 16:19:12.443588922 +0000 UTC m=+131.116123421" Jul 27 16:19:13.421351 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:13.421311 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x9964" event={"ID":"6a364884-ca85-47df-9f19-b2789c22c312","Type":"ContainerStarted","Data":"b375f8650407d4072be203c29031052ade992844bb3021c1490ac51919b42175"} Jul 27 16:19:13.422834 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:13.422768 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x9964" Jul 27 16:19:13.424289 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:13.424185 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mg2dr" event={"ID":"fc3e4156-6cfc-430a-8842-a9d801cfb9f0","Type":"ContainerStarted","Data":"60a6485595ef6a1453be35af98e69672458d47640312d7e6592359689348e41a"} Jul 27 16:19:13.424289 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:13.424246 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mg2dr" event={"ID":"fc3e4156-6cfc-430a-8842-a9d801cfb9f0","Type":"ContainerStarted","Data":"5ba8ce0529280a9cd527bbb868cdc2d13026e2eb35e282168c539c26d5868b9e"} Jul 27 16:19:13.428350 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:13.428325 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x9964" Jul 27 16:19:13.446916 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:13.446861 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x9964" podStartSLOduration=3.973006132 podStartE2EDuration="5.446844377s" totalImagesPullingTime="1.473838245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:13.444854068 +0000 UTC m=+132.117388567" watchObservedRunningTime="2026-07-27 16:19:13.446844377 +0000 UTC m=+132.119378877" Jul 27 16:19:13.447025 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:13.446998 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.284344532 podStartE2EDuration="9.446988592s" totalImagesPullingTime="5.143758414s" totalInitContainerRuntime="18.885646ms" isStatefulPod=false podCreationTimestamp="2026-07-27 16:19:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:12.480228581 +0000 UTC m=+131.152763081" watchObservedRunningTime="2026-07-27 16:19:13.446988592 +0000 UTC m=+132.119523128" Jul 27 16:19:13.468861 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:13.468722 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mg2dr" podStartSLOduration=130.304973548 podStartE2EDuration="2m11.468704093s" totalImagesPullingTime="1.163730545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:13.46810962 +0000 UTC m=+132.140644122" watchObservedRunningTime="2026-07-27 16:19:13.468704093 +0000 UTC m=+132.141238592" Jul 27 16:19:18.432014 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:18.431982 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6456df978d-5vg66" Jul 27 16:19:20.427145 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.427112 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55dd845c-l6pf5"] Jul 27 16:19:20.432447 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.432417 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.437723 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.437693 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bqpqf\"" Jul 27 16:19:20.438228 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.437973 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 27 16:19:20.438228 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.438082 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 27 16:19:20.438618 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.438598 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 27 16:19:20.438747 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.438728 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 27 16:19:20.439197 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.439150 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 27 16:19:20.447679 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.447641 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55dd845c-l6pf5"] Jul 27 16:19:20.514442 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.514402 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs2nv\" (UniqueName: \"kubernetes.io/projected/2f20447a-55b6-4173-b1ae-2c46465a21ee-kube-api-access-fs2nv\") pod \"console-55dd845c-l6pf5\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.514626 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.514451 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f20447a-55b6-4173-b1ae-2c46465a21ee-console-serving-cert\") pod \"console-55dd845c-l6pf5\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.514626 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.514490 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f20447a-55b6-4173-b1ae-2c46465a21ee-oauth-serving-cert\") pod \"console-55dd845c-l6pf5\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.514626 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.514560 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f20447a-55b6-4173-b1ae-2c46465a21ee-console-oauth-config\") pod \"console-55dd845c-l6pf5\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.514626 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.514578 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f20447a-55b6-4173-b1ae-2c46465a21ee-service-ca\") pod \"console-55dd845c-l6pf5\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.514626 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.514615 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f20447a-55b6-4173-b1ae-2c46465a21ee-console-config\") pod \"console-55dd845c-l6pf5\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.615357 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.615317 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs2nv\" (UniqueName: \"kubernetes.io/projected/2f20447a-55b6-4173-b1ae-2c46465a21ee-kube-api-access-fs2nv\") pod \"console-55dd845c-l6pf5\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.615357 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.615358 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f20447a-55b6-4173-b1ae-2c46465a21ee-console-serving-cert\") pod \"console-55dd845c-l6pf5\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.615596 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.615392 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f20447a-55b6-4173-b1ae-2c46465a21ee-oauth-serving-cert\") pod \"console-55dd845c-l6pf5\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.615596 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.615446 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f20447a-55b6-4173-b1ae-2c46465a21ee-console-oauth-config\") pod \"console-55dd845c-l6pf5\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.615596 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.615470 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f20447a-55b6-4173-b1ae-2c46465a21ee-service-ca\") pod \"console-55dd845c-l6pf5\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.615596 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.615506 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f20447a-55b6-4173-b1ae-2c46465a21ee-console-config\") pod \"console-55dd845c-l6pf5\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.616432 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.616374 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f20447a-55b6-4173-b1ae-2c46465a21ee-oauth-serving-cert\") pod \"console-55dd845c-l6pf5\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.616432 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.616408 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f20447a-55b6-4173-b1ae-2c46465a21ee-service-ca\") pod \"console-55dd845c-l6pf5\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.616958 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.616934 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f20447a-55b6-4173-b1ae-2c46465a21ee-console-config\") pod \"console-55dd845c-l6pf5\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.618383 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.618354 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f20447a-55b6-4173-b1ae-2c46465a21ee-console-oauth-config\") pod \"console-55dd845c-l6pf5\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.618513 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.618389 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f20447a-55b6-4173-b1ae-2c46465a21ee-console-serving-cert\") pod \"console-55dd845c-l6pf5\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.631092 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.631070 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs2nv\" (UniqueName: \"kubernetes.io/projected/2f20447a-55b6-4173-b1ae-2c46465a21ee-kube-api-access-fs2nv\") pod \"console-55dd845c-l6pf5\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.748538 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.748457 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:20.938873 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:20.938698 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55dd845c-l6pf5"] Jul 27 16:19:20.941739 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:19:20.941702 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f20447a_55b6_4173_b1ae_2c46465a21ee.slice/crio-cc312eadc480250d14ddcde5184bc8ea7e4c8fc413d540de276ba12439ac0676 WatchSource:0}: Error finding container cc312eadc480250d14ddcde5184bc8ea7e4c8fc413d540de276ba12439ac0676: Status 404 returned error can't find the container with id cc312eadc480250d14ddcde5184bc8ea7e4c8fc413d540de276ba12439ac0676 Jul 27 16:19:21.463118 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:21.463081 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55dd845c-l6pf5" event={"ID":"2f20447a-55b6-4173-b1ae-2c46465a21ee","Type":"ContainerStarted","Data":"cc312eadc480250d14ddcde5184bc8ea7e4c8fc413d540de276ba12439ac0676"} Jul 27 16:19:28.420034 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:28.420002 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:28.420453 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:28.420071 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:32.516549 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:32.516511 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55dd845c-l6pf5" event={"ID":"2f20447a-55b6-4173-b1ae-2c46465a21ee","Type":"ContainerStarted","Data":"d41e219cc5b9a52068d0ce07df3ef062a956f190ee246c3fd277c5ab637c5458"} Jul 27 16:19:32.518456 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:32.518425 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-l48kt" event={"ID":"223f7f5e-f930-4bfc-958d-9ec54d0a27a9","Type":"ContainerStarted","Data":"903d906835288792f84582a1ccea4a8b3fb66c4049d6131186a88c7d62ed839b"} Jul 27 16:19:32.518624 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:32.518607 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-l48kt" Jul 27 16:19:32.520329 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:32.520298 2550 generic.go:356] "Generic (PLEG): container finished" podID="3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6" containerID="0d97df4fba303c5c5f50d8ad5d5487a06664f985ca911aa77adacdbbe3e0fe88" exitCode=0 Jul 27 16:19:32.520444 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:32.520339 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wsq68" event={"ID":"3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6","Type":"ContainerDied","Data":"0d97df4fba303c5c5f50d8ad5d5487a06664f985ca911aa77adacdbbe3e0fe88"} Jul 27 16:19:32.520831 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:32.520727 2550 scope.go:117] "RemoveContainer" containerID="0d97df4fba303c5c5f50d8ad5d5487a06664f985ca911aa77adacdbbe3e0fe88" Jul 27 16:19:32.530995 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:32.530973 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-l48kt" Jul 27 16:19:32.540166 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:32.540112 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55dd845c-l6pf5" podStartSLOduration=1.446792404 podStartE2EDuration="12.540095069s" totalImagesPullingTime="11.093302665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:32.536961877 +0000 UTC m=+151.209496375" watchObservedRunningTime="2026-07-27 16:19:32.540095069 +0000 UTC m=+151.212629569" Jul 27 16:19:32.583605 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:32.583558 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-l48kt" podStartSLOduration=1.6026206090000001 podStartE2EDuration="25.583534233s" totalImagesPullingTime="23.980913624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:32.583342763 +0000 UTC m=+151.255877263" watchObservedRunningTime="2026-07-27 16:19:32.583534233 +0000 UTC m=+151.256068732" Jul 27 16:19:33.527889 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:33.527849 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wsq68" event={"ID":"3043ca1d-5bc5-42d3-8dcb-7a9211ba39d6","Type":"ContainerStarted","Data":"763f1ae3b102e98b85c3e7144a0a6924ccf24d7a161650d7aba8674f7e0ee2e2"} Jul 27 16:19:36.543767 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:36.543731 2550 generic.go:356] "Generic (PLEG): container finished" podID="359a50cb-e66a-4a77-8ae8-33bc152ad58f" containerID="4f3b52b2b6ce7ed8c722d1f1e9173245b56b894bc610526ae517609c1b9f3297" exitCode=0 Jul 27 16:19:36.544267 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:36.543783 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-dbfs5" event={"ID":"359a50cb-e66a-4a77-8ae8-33bc152ad58f","Type":"ContainerDied","Data":"4f3b52b2b6ce7ed8c722d1f1e9173245b56b894bc610526ae517609c1b9f3297"} Jul 27 16:19:36.544267 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:36.544229 2550 scope.go:117] "RemoveContainer" containerID="4f3b52b2b6ce7ed8c722d1f1e9173245b56b894bc610526ae517609c1b9f3297" Jul 27 16:19:37.550811 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:37.550776 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-dbfs5" event={"ID":"359a50cb-e66a-4a77-8ae8-33bc152ad58f","Type":"ContainerStarted","Data":"4ba1c7da09fdaf26059048f34d6ccf6712311c0ee483cac478d3c20a040e4bc5"} Jul 27 16:19:40.749455 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:40.749414 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:40.749949 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:40.749489 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:40.755136 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:40.755114 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:41.506462 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:41.506424 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55dd845c-l6pf5"] Jul 27 16:19:41.572931 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:41.572886 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:19:48.425853 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:48.425822 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:19:48.430795 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:19:48.430772 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-76d9d7c7f8-5rn2l" Jul 27 16:20:07.592323 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:07.592262 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55dd845c-l6pf5" podUID="2f20447a-55b6-4173-b1ae-2c46465a21ee" containerName="console" containerID="cri-o://d41e219cc5b9a52068d0ce07df3ef062a956f190ee246c3fd277c5ab637c5458" gracePeriod=15 Jul 27 16:20:07.861932 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:07.861854 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:20:07.929984 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:07.929956 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f20447a-55b6-4173-b1ae-2c46465a21ee-service-ca\") pod \"2f20447a-55b6-4173-b1ae-2c46465a21ee\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " Jul 27 16:20:07.930135 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:07.930000 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f20447a-55b6-4173-b1ae-2c46465a21ee-console-oauth-config\") pod \"2f20447a-55b6-4173-b1ae-2c46465a21ee\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " Jul 27 16:20:07.930135 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:07.930026 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f20447a-55b6-4173-b1ae-2c46465a21ee-oauth-serving-cert\") pod \"2f20447a-55b6-4173-b1ae-2c46465a21ee\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " Jul 27 16:20:07.930135 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:07.930043 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f20447a-55b6-4173-b1ae-2c46465a21ee-console-serving-cert\") pod \"2f20447a-55b6-4173-b1ae-2c46465a21ee\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " Jul 27 16:20:07.930135 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:07.930071 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fs2nv\" (UniqueName: \"kubernetes.io/projected/2f20447a-55b6-4173-b1ae-2c46465a21ee-kube-api-access-fs2nv\") pod \"2f20447a-55b6-4173-b1ae-2c46465a21ee\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " Jul 27 16:20:07.930135 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:07.930097 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f20447a-55b6-4173-b1ae-2c46465a21ee-console-config\") pod \"2f20447a-55b6-4173-b1ae-2c46465a21ee\" (UID: \"2f20447a-55b6-4173-b1ae-2c46465a21ee\") " Jul 27 16:20:07.930476 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:07.930379 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f20447a-55b6-4173-b1ae-2c46465a21ee-service-ca" (OuterVolumeSpecName: "service-ca") pod "2f20447a-55b6-4173-b1ae-2c46465a21ee" (UID: "2f20447a-55b6-4173-b1ae-2c46465a21ee"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:20:07.930553 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:07.930477 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f20447a-55b6-4173-b1ae-2c46465a21ee-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2f20447a-55b6-4173-b1ae-2c46465a21ee" (UID: "2f20447a-55b6-4173-b1ae-2c46465a21ee"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:20:07.930822 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:07.930772 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f20447a-55b6-4173-b1ae-2c46465a21ee-console-config" (OuterVolumeSpecName: "console-config") pod "2f20447a-55b6-4173-b1ae-2c46465a21ee" (UID: "2f20447a-55b6-4173-b1ae-2c46465a21ee"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:20:07.932496 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:07.932468 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f20447a-55b6-4173-b1ae-2c46465a21ee-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2f20447a-55b6-4173-b1ae-2c46465a21ee" (UID: "2f20447a-55b6-4173-b1ae-2c46465a21ee"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 16:20:07.932712 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:07.932691 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f20447a-55b6-4173-b1ae-2c46465a21ee-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2f20447a-55b6-4173-b1ae-2c46465a21ee" (UID: "2f20447a-55b6-4173-b1ae-2c46465a21ee"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 16:20:07.932803 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:07.932717 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f20447a-55b6-4173-b1ae-2c46465a21ee-kube-api-access-fs2nv" (OuterVolumeSpecName: "kube-api-access-fs2nv") pod "2f20447a-55b6-4173-b1ae-2c46465a21ee" (UID: "2f20447a-55b6-4173-b1ae-2c46465a21ee"). InnerVolumeSpecName "kube-api-access-fs2nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:20:08.031062 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:08.031033 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs2nv\" (UniqueName: \"kubernetes.io/projected/2f20447a-55b6-4173-b1ae-2c46465a21ee-kube-api-access-fs2nv\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:20:08.031062 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:08.031058 2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f20447a-55b6-4173-b1ae-2c46465a21ee-console-config\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:20:08.031062 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:08.031068 2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f20447a-55b6-4173-b1ae-2c46465a21ee-service-ca\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:20:08.031276 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:08.031076 2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f20447a-55b6-4173-b1ae-2c46465a21ee-console-oauth-config\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:20:08.031276 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:08.031084 2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f20447a-55b6-4173-b1ae-2c46465a21ee-oauth-serving-cert\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:20:08.031276 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:08.031092 2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f20447a-55b6-4173-b1ae-2c46465a21ee-console-serving-cert\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:20:08.684483 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:08.684443 2550 generic.go:356] "Generic (PLEG): container finished" podID="2f20447a-55b6-4173-b1ae-2c46465a21ee" containerID="d41e219cc5b9a52068d0ce07df3ef062a956f190ee246c3fd277c5ab637c5458" exitCode=0 Jul 27 16:20:08.684864 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:08.684493 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55dd845c-l6pf5" Jul 27 16:20:08.684864 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:08.684525 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55dd845c-l6pf5" event={"ID":"2f20447a-55b6-4173-b1ae-2c46465a21ee","Type":"ContainerDied","Data":"d41e219cc5b9a52068d0ce07df3ef062a956f190ee246c3fd277c5ab637c5458"} Jul 27 16:20:08.684864 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:08.684575 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55dd845c-l6pf5" event={"ID":"2f20447a-55b6-4173-b1ae-2c46465a21ee","Type":"ContainerDied","Data":"cc312eadc480250d14ddcde5184bc8ea7e4c8fc413d540de276ba12439ac0676"} Jul 27 16:20:08.684864 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:08.684596 2550 scope.go:117] "RemoveContainer" containerID="d41e219cc5b9a52068d0ce07df3ef062a956f190ee246c3fd277c5ab637c5458" Jul 27 16:20:08.697159 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:08.697138 2550 scope.go:117] "RemoveContainer" containerID="d41e219cc5b9a52068d0ce07df3ef062a956f190ee246c3fd277c5ab637c5458" Jul 27 16:20:08.697426 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:20:08.697404 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d41e219cc5b9a52068d0ce07df3ef062a956f190ee246c3fd277c5ab637c5458\": container with ID starting with d41e219cc5b9a52068d0ce07df3ef062a956f190ee246c3fd277c5ab637c5458 not found: ID does not exist" containerID="d41e219cc5b9a52068d0ce07df3ef062a956f190ee246c3fd277c5ab637c5458" Jul 27 16:20:08.697500 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:08.697433 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d41e219cc5b9a52068d0ce07df3ef062a956f190ee246c3fd277c5ab637c5458"} err="failed to get container status \"d41e219cc5b9a52068d0ce07df3ef062a956f190ee246c3fd277c5ab637c5458\": rpc error: code = NotFound desc = could not find container \"d41e219cc5b9a52068d0ce07df3ef062a956f190ee246c3fd277c5ab637c5458\": container with ID starting with d41e219cc5b9a52068d0ce07df3ef062a956f190ee246c3fd277c5ab637c5458 not found: ID does not exist" Jul 27 16:20:08.717137 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:08.717113 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55dd845c-l6pf5"] Jul 27 16:20:08.722959 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:08.722940 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55dd845c-l6pf5"] Jul 27 16:20:09.848360 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:09.848330 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f20447a-55b6-4173-b1ae-2c46465a21ee" path="/var/lib/kubelet/pods/2f20447a-55b6-4173-b1ae-2c46465a21ee/volumes" Jul 27 16:20:24.176041 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:24.175986 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 16:20:24.176505 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:24.176279 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="alertmanager" containerID="cri-o://f56ac2d66adab968a35ba13cf947f1a6a590ce145b855ca6b1bc6260dd6e6d42" gracePeriod=120 Jul 27 16:20:24.176505 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:24.176347 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="kube-rbac-proxy-metric" containerID="cri-o://e8a63c095ad34c28d564870d8ebdc6b41f80363c43d6effbf489a0804ce9093b" gracePeriod=120 Jul 27 16:20:24.176505 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:24.176389 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="config-reloader" containerID="cri-o://d68e31e472598c7264a395cd75095e7d1781d94b863463c67071537eac60dee7" gracePeriod=120 Jul 27 16:20:24.176505 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:24.176387 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="kube-rbac-proxy-web" containerID="cri-o://b0a76cedbd833631d03b2af7b54cd16cce8b8ce309c32b7bab27f8e0c2a5354c" gracePeriod=120 Jul 27 16:20:24.176505 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:24.176481 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="prom-label-proxy" containerID="cri-o://94ed3c68bae59e97ef7049a2c3fc81da5d97720db15ef867d2a63e062ba98100" gracePeriod=120 Jul 27 16:20:24.176505 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:24.176479 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="kube-rbac-proxy" containerID="cri-o://1cb22c6beda5be30861513e79b0baed7ea35efd4ed737f4d0ffd6a985eedb79e" gracePeriod=120 Jul 27 16:20:24.751267 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:24.751236 2550 generic.go:356] "Generic (PLEG): container finished" podID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerID="94ed3c68bae59e97ef7049a2c3fc81da5d97720db15ef867d2a63e062ba98100" exitCode=0 Jul 27 16:20:24.751267 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:24.751261 2550 generic.go:356] "Generic (PLEG): container finished" podID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerID="1cb22c6beda5be30861513e79b0baed7ea35efd4ed737f4d0ffd6a985eedb79e" exitCode=0 Jul 27 16:20:24.751267 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:24.751267 2550 generic.go:356] "Generic (PLEG): container finished" podID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerID="d68e31e472598c7264a395cd75095e7d1781d94b863463c67071537eac60dee7" exitCode=0 Jul 27 16:20:24.751267 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:24.751273 2550 generic.go:356] "Generic (PLEG): container finished" podID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerID="f56ac2d66adab968a35ba13cf947f1a6a590ce145b855ca6b1bc6260dd6e6d42" exitCode=0 Jul 27 16:20:24.751547 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:24.751307 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf37d2a1-2ccb-49e7-9a33-80b61390d132","Type":"ContainerDied","Data":"94ed3c68bae59e97ef7049a2c3fc81da5d97720db15ef867d2a63e062ba98100"} Jul 27 16:20:24.751547 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:24.751394 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf37d2a1-2ccb-49e7-9a33-80b61390d132","Type":"ContainerDied","Data":"1cb22c6beda5be30861513e79b0baed7ea35efd4ed737f4d0ffd6a985eedb79e"} Jul 27 16:20:24.751547 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:24.751421 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf37d2a1-2ccb-49e7-9a33-80b61390d132","Type":"ContainerDied","Data":"d68e31e472598c7264a395cd75095e7d1781d94b863463c67071537eac60dee7"} Jul 27 16:20:24.751547 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:24.751445 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf37d2a1-2ccb-49e7-9a33-80b61390d132","Type":"ContainerDied","Data":"f56ac2d66adab968a35ba13cf947f1a6a590ce145b855ca6b1bc6260dd6e6d42"} Jul 27 16:20:25.419453 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.419429 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:25.479228 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.479149 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-cluster-tls-config\") pod \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " Jul 27 16:20:25.479228 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.479217 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-main-tls\") pod \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " Jul 27 16:20:25.479456 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.479242 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-kube-rbac-proxy-web\") pod \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " Jul 27 16:20:25.479456 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.479265 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btt2p\" (UniqueName: \"kubernetes.io/projected/bf37d2a1-2ccb-49e7-9a33-80b61390d132-kube-api-access-btt2p\") pod \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " Jul 27 16:20:25.479456 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.479283 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-config-volume\") pod \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " Jul 27 16:20:25.479456 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.479309 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf37d2a1-2ccb-49e7-9a33-80b61390d132-alertmanager-trusted-ca-bundle\") pod \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " Jul 27 16:20:25.479456 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.479335 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf37d2a1-2ccb-49e7-9a33-80b61390d132-config-out\") pod \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " Jul 27 16:20:25.479456 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.479357 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf37d2a1-2ccb-49e7-9a33-80b61390d132-metrics-client-ca\") pod \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " Jul 27 16:20:25.479456 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.479389 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-kube-rbac-proxy-metric\") pod \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " Jul 27 16:20:25.479456 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.479407 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf37d2a1-2ccb-49e7-9a33-80b61390d132-tls-assets\") pod \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " Jul 27 16:20:25.479456 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.479436 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-web-config\") pod \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " Jul 27 16:20:25.479892 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.479468 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bf37d2a1-2ccb-49e7-9a33-80b61390d132-alertmanager-main-db\") pod \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " Jul 27 16:20:25.479892 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.479486 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-kube-rbac-proxy\") pod \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\" (UID: \"bf37d2a1-2ccb-49e7-9a33-80b61390d132\") " Jul 27 16:20:25.480410 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.480336 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf37d2a1-2ccb-49e7-9a33-80b61390d132-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "bf37d2a1-2ccb-49e7-9a33-80b61390d132" (UID: "bf37d2a1-2ccb-49e7-9a33-80b61390d132"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:20:25.480779 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.480754 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf37d2a1-2ccb-49e7-9a33-80b61390d132-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "bf37d2a1-2ccb-49e7-9a33-80b61390d132" (UID: "bf37d2a1-2ccb-49e7-9a33-80b61390d132"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:20:25.484148 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.484095 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf37d2a1-2ccb-49e7-9a33-80b61390d132-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "bf37d2a1-2ccb-49e7-9a33-80b61390d132" (UID: "bf37d2a1-2ccb-49e7-9a33-80b61390d132"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:20:25.484572 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.484531 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf37d2a1-2ccb-49e7-9a33-80b61390d132-config-out" (OuterVolumeSpecName: "config-out") pod "bf37d2a1-2ccb-49e7-9a33-80b61390d132" (UID: "bf37d2a1-2ccb-49e7-9a33-80b61390d132"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:20:25.484674 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.484588 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "bf37d2a1-2ccb-49e7-9a33-80b61390d132" (UID: "bf37d2a1-2ccb-49e7-9a33-80b61390d132"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 16:20:25.484872 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.484845 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-config-volume" (OuterVolumeSpecName: "config-volume") pod "bf37d2a1-2ccb-49e7-9a33-80b61390d132" (UID: "bf37d2a1-2ccb-49e7-9a33-80b61390d132"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 16:20:25.485329 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.485297 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf37d2a1-2ccb-49e7-9a33-80b61390d132-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "bf37d2a1-2ccb-49e7-9a33-80b61390d132" (UID: "bf37d2a1-2ccb-49e7-9a33-80b61390d132"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:20:25.485794 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.485759 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "bf37d2a1-2ccb-49e7-9a33-80b61390d132" (UID: "bf37d2a1-2ccb-49e7-9a33-80b61390d132"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 16:20:25.486132 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.486108 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "bf37d2a1-2ccb-49e7-9a33-80b61390d132" (UID: "bf37d2a1-2ccb-49e7-9a33-80b61390d132"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 16:20:25.486806 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.486777 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "bf37d2a1-2ccb-49e7-9a33-80b61390d132" (UID: "bf37d2a1-2ccb-49e7-9a33-80b61390d132"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 16:20:25.487889 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.487858 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf37d2a1-2ccb-49e7-9a33-80b61390d132-kube-api-access-btt2p" (OuterVolumeSpecName: "kube-api-access-btt2p") pod "bf37d2a1-2ccb-49e7-9a33-80b61390d132" (UID: "bf37d2a1-2ccb-49e7-9a33-80b61390d132"). InnerVolumeSpecName "kube-api-access-btt2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:20:25.489097 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.489079 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "bf37d2a1-2ccb-49e7-9a33-80b61390d132" (UID: "bf37d2a1-2ccb-49e7-9a33-80b61390d132"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 16:20:25.495403 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.495365 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-web-config" (OuterVolumeSpecName: "web-config") pod "bf37d2a1-2ccb-49e7-9a33-80b61390d132" (UID: "bf37d2a1-2ccb-49e7-9a33-80b61390d132"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 16:20:25.581015 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.580967 2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-main-tls\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:20:25.581015 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.581011 2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:20:25.581015 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.581027 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btt2p\" (UniqueName: \"kubernetes.io/projected/bf37d2a1-2ccb-49e7-9a33-80b61390d132-kube-api-access-btt2p\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:20:25.581254 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.581041 2550 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-config-volume\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:20:25.581254 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.581053 2550 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf37d2a1-2ccb-49e7-9a33-80b61390d132-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:20:25.581254 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.581064 2550 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf37d2a1-2ccb-49e7-9a33-80b61390d132-config-out\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:20:25.581254 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.581074 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf37d2a1-2ccb-49e7-9a33-80b61390d132-metrics-client-ca\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:20:25.581254 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.581085 2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:20:25.581254 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.581097 2550 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf37d2a1-2ccb-49e7-9a33-80b61390d132-tls-assets\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:20:25.581254 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.581109 2550 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-web-config\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:20:25.581254 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.581119 2550 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bf37d2a1-2ccb-49e7-9a33-80b61390d132-alertmanager-main-db\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:20:25.581254 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.581130 2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:20:25.581254 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.581142 2550 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf37d2a1-2ccb-49e7-9a33-80b61390d132-cluster-tls-config\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:20:25.760695 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.760603 2550 generic.go:356] "Generic (PLEG): container finished" podID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerID="e8a63c095ad34c28d564870d8ebdc6b41f80363c43d6effbf489a0804ce9093b" exitCode=0 Jul 27 16:20:25.760695 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.760628 2550 generic.go:356] "Generic (PLEG): container finished" podID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerID="b0a76cedbd833631d03b2af7b54cd16cce8b8ce309c32b7bab27f8e0c2a5354c" exitCode=0 Jul 27 16:20:25.760695 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.760672 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf37d2a1-2ccb-49e7-9a33-80b61390d132","Type":"ContainerDied","Data":"e8a63c095ad34c28d564870d8ebdc6b41f80363c43d6effbf489a0804ce9093b"} Jul 27 16:20:25.760969 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.760746 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:25.760969 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.760801 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf37d2a1-2ccb-49e7-9a33-80b61390d132","Type":"ContainerDied","Data":"b0a76cedbd833631d03b2af7b54cd16cce8b8ce309c32b7bab27f8e0c2a5354c"} Jul 27 16:20:25.760969 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.760845 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bf37d2a1-2ccb-49e7-9a33-80b61390d132","Type":"ContainerDied","Data":"aa7a5e9835d250bd9479c5837f102fc94686ee9562beef3f92e9506f9ad033bd"} Jul 27 16:20:25.760969 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.760866 2550 scope.go:117] "RemoveContainer" containerID="94ed3c68bae59e97ef7049a2c3fc81da5d97720db15ef867d2a63e062ba98100" Jul 27 16:20:25.769392 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.769378 2550 scope.go:117] "RemoveContainer" containerID="e8a63c095ad34c28d564870d8ebdc6b41f80363c43d6effbf489a0804ce9093b" Jul 27 16:20:25.776404 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.776388 2550 scope.go:117] "RemoveContainer" containerID="1cb22c6beda5be30861513e79b0baed7ea35efd4ed737f4d0ffd6a985eedb79e" Jul 27 16:20:25.782974 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.782952 2550 scope.go:117] "RemoveContainer" containerID="b0a76cedbd833631d03b2af7b54cd16cce8b8ce309c32b7bab27f8e0c2a5354c" Jul 27 16:20:25.789256 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.788698 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 16:20:25.790620 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.790604 2550 scope.go:117] "RemoveContainer" containerID="d68e31e472598c7264a395cd75095e7d1781d94b863463c67071537eac60dee7" Jul 27 16:20:25.797648 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.797627 2550 scope.go:117] "RemoveContainer" containerID="f56ac2d66adab968a35ba13cf947f1a6a590ce145b855ca6b1bc6260dd6e6d42" Jul 27 16:20:25.799095 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.799077 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 16:20:25.804319 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804300 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d5c4dfbcd-8h8ft"] Jul 27 16:20:25.804580 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804569 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="kube-rbac-proxy-metric" Jul 27 16:20:25.804628 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804581 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="kube-rbac-proxy-metric" Jul 27 16:20:25.804628 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804588 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="prom-label-proxy" Jul 27 16:20:25.804628 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804593 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="prom-label-proxy" Jul 27 16:20:25.804628 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804599 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f20447a-55b6-4173-b1ae-2c46465a21ee" containerName="console" Jul 27 16:20:25.804628 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804604 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f20447a-55b6-4173-b1ae-2c46465a21ee" containerName="console" Jul 27 16:20:25.804628 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804612 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="kube-rbac-proxy-web" Jul 27 16:20:25.804628 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804617 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="kube-rbac-proxy-web" Jul 27 16:20:25.804628 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804629 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="init-config-reloader" Jul 27 16:20:25.804882 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804634 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="init-config-reloader" Jul 27 16:20:25.804882 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804641 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="config-reloader" Jul 27 16:20:25.804882 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804646 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="config-reloader" Jul 27 16:20:25.804882 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804653 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="alertmanager" Jul 27 16:20:25.804882 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804658 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="alertmanager" Jul 27 16:20:25.804882 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804664 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="kube-rbac-proxy" Jul 27 16:20:25.804882 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804670 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="kube-rbac-proxy" Jul 27 16:20:25.804882 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804713 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="alertmanager" Jul 27 16:20:25.804882 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804721 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="prom-label-proxy" Jul 27 16:20:25.804882 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804726 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f20447a-55b6-4173-b1ae-2c46465a21ee" containerName="console" Jul 27 16:20:25.804882 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804732 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="kube-rbac-proxy-web" Jul 27 16:20:25.804882 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804738 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="config-reloader" Jul 27 16:20:25.804882 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804745 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="kube-rbac-proxy-metric" Jul 27 16:20:25.804882 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804750 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" containerName="kube-rbac-proxy" Jul 27 16:20:25.805392 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.804968 2550 scope.go:117] "RemoveContainer" containerID="a7f880875ba319f4bd5d82b5ff634aaf91dbd4cc5b6565e3a493bc82f14da5be" Jul 27 16:20:25.807788 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.807772 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.812421 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.812406 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 27 16:20:25.812631 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.812616 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 27 16:20:25.812631 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.812616 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 27 16:20:25.812824 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.812666 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 27 16:20:25.812824 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.812791 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bqpqf\"" Jul 27 16:20:25.813038 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.813016 2550 scope.go:117] "RemoveContainer" containerID="94ed3c68bae59e97ef7049a2c3fc81da5d97720db15ef867d2a63e062ba98100" Jul 27 16:20:25.813147 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.813045 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 27 16:20:25.813266 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:20:25.813223 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94ed3c68bae59e97ef7049a2c3fc81da5d97720db15ef867d2a63e062ba98100\": container with ID starting with 94ed3c68bae59e97ef7049a2c3fc81da5d97720db15ef867d2a63e062ba98100 not found: ID does not exist" containerID="94ed3c68bae59e97ef7049a2c3fc81da5d97720db15ef867d2a63e062ba98100" Jul 27 16:20:25.813366 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.813259 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94ed3c68bae59e97ef7049a2c3fc81da5d97720db15ef867d2a63e062ba98100"} err="failed to get container status \"94ed3c68bae59e97ef7049a2c3fc81da5d97720db15ef867d2a63e062ba98100\": rpc error: code = NotFound desc = could not find container \"94ed3c68bae59e97ef7049a2c3fc81da5d97720db15ef867d2a63e062ba98100\": container with ID starting with 94ed3c68bae59e97ef7049a2c3fc81da5d97720db15ef867d2a63e062ba98100 not found: ID does not exist" Jul 27 16:20:25.813366 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.813283 2550 scope.go:117] "RemoveContainer" containerID="e8a63c095ad34c28d564870d8ebdc6b41f80363c43d6effbf489a0804ce9093b" Jul 27 16:20:25.813581 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:20:25.813560 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a63c095ad34c28d564870d8ebdc6b41f80363c43d6effbf489a0804ce9093b\": container with ID starting with e8a63c095ad34c28d564870d8ebdc6b41f80363c43d6effbf489a0804ce9093b not found: ID does not exist" containerID="e8a63c095ad34c28d564870d8ebdc6b41f80363c43d6effbf489a0804ce9093b" Jul 27 16:20:25.813631 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.813587 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a63c095ad34c28d564870d8ebdc6b41f80363c43d6effbf489a0804ce9093b"} err="failed to get container status \"e8a63c095ad34c28d564870d8ebdc6b41f80363c43d6effbf489a0804ce9093b\": rpc error: code = NotFound desc = could not find container \"e8a63c095ad34c28d564870d8ebdc6b41f80363c43d6effbf489a0804ce9093b\": container with ID starting with e8a63c095ad34c28d564870d8ebdc6b41f80363c43d6effbf489a0804ce9093b not found: ID does not exist" Jul 27 16:20:25.813631 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.813627 2550 scope.go:117] "RemoveContainer" containerID="1cb22c6beda5be30861513e79b0baed7ea35efd4ed737f4d0ffd6a985eedb79e" Jul 27 16:20:25.813982 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:20:25.813880 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cb22c6beda5be30861513e79b0baed7ea35efd4ed737f4d0ffd6a985eedb79e\": container with ID starting with 1cb22c6beda5be30861513e79b0baed7ea35efd4ed737f4d0ffd6a985eedb79e not found: ID does not exist" containerID="1cb22c6beda5be30861513e79b0baed7ea35efd4ed737f4d0ffd6a985eedb79e" Jul 27 16:20:25.813982 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.813929 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cb22c6beda5be30861513e79b0baed7ea35efd4ed737f4d0ffd6a985eedb79e"} err="failed to get container status \"1cb22c6beda5be30861513e79b0baed7ea35efd4ed737f4d0ffd6a985eedb79e\": rpc error: code = NotFound desc = could not find container \"1cb22c6beda5be30861513e79b0baed7ea35efd4ed737f4d0ffd6a985eedb79e\": container with ID starting with 1cb22c6beda5be30861513e79b0baed7ea35efd4ed737f4d0ffd6a985eedb79e not found: ID does not exist" Jul 27 16:20:25.813982 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.813950 2550 scope.go:117] "RemoveContainer" containerID="b0a76cedbd833631d03b2af7b54cd16cce8b8ce309c32b7bab27f8e0c2a5354c" Jul 27 16:20:25.814239 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:20:25.814208 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a76cedbd833631d03b2af7b54cd16cce8b8ce309c32b7bab27f8e0c2a5354c\": container with ID starting with b0a76cedbd833631d03b2af7b54cd16cce8b8ce309c32b7bab27f8e0c2a5354c not found: ID does not exist" containerID="b0a76cedbd833631d03b2af7b54cd16cce8b8ce309c32b7bab27f8e0c2a5354c" Jul 27 16:20:25.814321 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.814248 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a76cedbd833631d03b2af7b54cd16cce8b8ce309c32b7bab27f8e0c2a5354c"} err="failed to get container status \"b0a76cedbd833631d03b2af7b54cd16cce8b8ce309c32b7bab27f8e0c2a5354c\": rpc error: code = NotFound desc = could not find container \"b0a76cedbd833631d03b2af7b54cd16cce8b8ce309c32b7bab27f8e0c2a5354c\": container with ID starting with b0a76cedbd833631d03b2af7b54cd16cce8b8ce309c32b7bab27f8e0c2a5354c not found: ID does not exist" Jul 27 16:20:25.814321 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.814273 2550 scope.go:117] "RemoveContainer" containerID="d68e31e472598c7264a395cd75095e7d1781d94b863463c67071537eac60dee7" Jul 27 16:20:25.814610 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:20:25.814511 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d68e31e472598c7264a395cd75095e7d1781d94b863463c67071537eac60dee7\": container with ID starting with d68e31e472598c7264a395cd75095e7d1781d94b863463c67071537eac60dee7 not found: ID does not exist" containerID="d68e31e472598c7264a395cd75095e7d1781d94b863463c67071537eac60dee7" Jul 27 16:20:25.814610 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.814535 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68e31e472598c7264a395cd75095e7d1781d94b863463c67071537eac60dee7"} err="failed to get container status \"d68e31e472598c7264a395cd75095e7d1781d94b863463c67071537eac60dee7\": rpc error: code = NotFound desc = could not find container \"d68e31e472598c7264a395cd75095e7d1781d94b863463c67071537eac60dee7\": container with ID starting with d68e31e472598c7264a395cd75095e7d1781d94b863463c67071537eac60dee7 not found: ID does not exist" Jul 27 16:20:25.814610 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.814554 2550 scope.go:117] "RemoveContainer" containerID="f56ac2d66adab968a35ba13cf947f1a6a590ce145b855ca6b1bc6260dd6e6d42" Jul 27 16:20:25.814961 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:20:25.814881 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56ac2d66adab968a35ba13cf947f1a6a590ce145b855ca6b1bc6260dd6e6d42\": container with ID starting with f56ac2d66adab968a35ba13cf947f1a6a590ce145b855ca6b1bc6260dd6e6d42 not found: ID does not exist" containerID="f56ac2d66adab968a35ba13cf947f1a6a590ce145b855ca6b1bc6260dd6e6d42" Jul 27 16:20:25.814961 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.814931 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56ac2d66adab968a35ba13cf947f1a6a590ce145b855ca6b1bc6260dd6e6d42"} err="failed to get container status \"f56ac2d66adab968a35ba13cf947f1a6a590ce145b855ca6b1bc6260dd6e6d42\": rpc error: code = NotFound desc = could not find container \"f56ac2d66adab968a35ba13cf947f1a6a590ce145b855ca6b1bc6260dd6e6d42\": container with ID starting with f56ac2d66adab968a35ba13cf947f1a6a590ce145b855ca6b1bc6260dd6e6d42 not found: ID does not exist" Jul 27 16:20:25.814961 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.814959 2550 scope.go:117] "RemoveContainer" containerID="a7f880875ba319f4bd5d82b5ff634aaf91dbd4cc5b6565e3a493bc82f14da5be" Jul 27 16:20:25.815247 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:20:25.815214 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f880875ba319f4bd5d82b5ff634aaf91dbd4cc5b6565e3a493bc82f14da5be\": container with ID starting with a7f880875ba319f4bd5d82b5ff634aaf91dbd4cc5b6565e3a493bc82f14da5be not found: ID does not exist" containerID="a7f880875ba319f4bd5d82b5ff634aaf91dbd4cc5b6565e3a493bc82f14da5be" Jul 27 16:20:25.815318 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.815240 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f880875ba319f4bd5d82b5ff634aaf91dbd4cc5b6565e3a493bc82f14da5be"} err="failed to get container status \"a7f880875ba319f4bd5d82b5ff634aaf91dbd4cc5b6565e3a493bc82f14da5be\": rpc error: code = NotFound desc = could not find container \"a7f880875ba319f4bd5d82b5ff634aaf91dbd4cc5b6565e3a493bc82f14da5be\": container with ID starting with a7f880875ba319f4bd5d82b5ff634aaf91dbd4cc5b6565e3a493bc82f14da5be not found: ID does not exist" Jul 27 16:20:25.815318 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.815259 2550 scope.go:117] "RemoveContainer" containerID="94ed3c68bae59e97ef7049a2c3fc81da5d97720db15ef867d2a63e062ba98100" Jul 27 16:20:25.815572 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.815549 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94ed3c68bae59e97ef7049a2c3fc81da5d97720db15ef867d2a63e062ba98100"} err="failed to get container status \"94ed3c68bae59e97ef7049a2c3fc81da5d97720db15ef867d2a63e062ba98100\": rpc error: code = NotFound desc = could not find container \"94ed3c68bae59e97ef7049a2c3fc81da5d97720db15ef867d2a63e062ba98100\": container with ID starting with 94ed3c68bae59e97ef7049a2c3fc81da5d97720db15ef867d2a63e062ba98100 not found: ID does not exist" Jul 27 16:20:25.815572 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.815568 2550 scope.go:117] "RemoveContainer" containerID="e8a63c095ad34c28d564870d8ebdc6b41f80363c43d6effbf489a0804ce9093b" Jul 27 16:20:25.815801 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.815785 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a63c095ad34c28d564870d8ebdc6b41f80363c43d6effbf489a0804ce9093b"} err="failed to get container status \"e8a63c095ad34c28d564870d8ebdc6b41f80363c43d6effbf489a0804ce9093b\": rpc error: code = NotFound desc = could not find container \"e8a63c095ad34c28d564870d8ebdc6b41f80363c43d6effbf489a0804ce9093b\": container with ID starting with e8a63c095ad34c28d564870d8ebdc6b41f80363c43d6effbf489a0804ce9093b not found: ID does not exist" Jul 27 16:20:25.815891 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.815802 2550 scope.go:117] "RemoveContainer" containerID="1cb22c6beda5be30861513e79b0baed7ea35efd4ed737f4d0ffd6a985eedb79e" Jul 27 16:20:25.816062 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.816040 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cb22c6beda5be30861513e79b0baed7ea35efd4ed737f4d0ffd6a985eedb79e"} err="failed to get container status \"1cb22c6beda5be30861513e79b0baed7ea35efd4ed737f4d0ffd6a985eedb79e\": rpc error: code = NotFound desc = could not find container \"1cb22c6beda5be30861513e79b0baed7ea35efd4ed737f4d0ffd6a985eedb79e\": container with ID starting with 1cb22c6beda5be30861513e79b0baed7ea35efd4ed737f4d0ffd6a985eedb79e not found: ID does not exist" Jul 27 16:20:25.816150 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.816066 2550 scope.go:117] "RemoveContainer" containerID="b0a76cedbd833631d03b2af7b54cd16cce8b8ce309c32b7bab27f8e0c2a5354c" Jul 27 16:20:25.816324 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.816304 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a76cedbd833631d03b2af7b54cd16cce8b8ce309c32b7bab27f8e0c2a5354c"} err="failed to get container status \"b0a76cedbd833631d03b2af7b54cd16cce8b8ce309c32b7bab27f8e0c2a5354c\": rpc error: code = NotFound desc = could not find container \"b0a76cedbd833631d03b2af7b54cd16cce8b8ce309c32b7bab27f8e0c2a5354c\": container with ID starting with b0a76cedbd833631d03b2af7b54cd16cce8b8ce309c32b7bab27f8e0c2a5354c not found: ID does not exist" Jul 27 16:20:25.816324 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.816322 2550 scope.go:117] "RemoveContainer" containerID="d68e31e472598c7264a395cd75095e7d1781d94b863463c67071537eac60dee7" Jul 27 16:20:25.816567 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.816547 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68e31e472598c7264a395cd75095e7d1781d94b863463c67071537eac60dee7"} err="failed to get container status \"d68e31e472598c7264a395cd75095e7d1781d94b863463c67071537eac60dee7\": rpc error: code = NotFound desc = could not find container \"d68e31e472598c7264a395cd75095e7d1781d94b863463c67071537eac60dee7\": container with ID starting with d68e31e472598c7264a395cd75095e7d1781d94b863463c67071537eac60dee7 not found: ID does not exist" Jul 27 16:20:25.816645 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.816569 2550 scope.go:117] "RemoveContainer" containerID="f56ac2d66adab968a35ba13cf947f1a6a590ce145b855ca6b1bc6260dd6e6d42" Jul 27 16:20:25.816797 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.816769 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56ac2d66adab968a35ba13cf947f1a6a590ce145b855ca6b1bc6260dd6e6d42"} err="failed to get container status \"f56ac2d66adab968a35ba13cf947f1a6a590ce145b855ca6b1bc6260dd6e6d42\": rpc error: code = NotFound desc = could not find container \"f56ac2d66adab968a35ba13cf947f1a6a590ce145b855ca6b1bc6260dd6e6d42\": container with ID starting with f56ac2d66adab968a35ba13cf947f1a6a590ce145b855ca6b1bc6260dd6e6d42 not found: ID does not exist" Jul 27 16:20:25.816797 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.816787 2550 scope.go:117] "RemoveContainer" containerID="a7f880875ba319f4bd5d82b5ff634aaf91dbd4cc5b6565e3a493bc82f14da5be" Jul 27 16:20:25.817074 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.817052 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f880875ba319f4bd5d82b5ff634aaf91dbd4cc5b6565e3a493bc82f14da5be"} err="failed to get container status \"a7f880875ba319f4bd5d82b5ff634aaf91dbd4cc5b6565e3a493bc82f14da5be\": rpc error: code = NotFound desc = could not find container \"a7f880875ba319f4bd5d82b5ff634aaf91dbd4cc5b6565e3a493bc82f14da5be\": container with ID starting with a7f880875ba319f4bd5d82b5ff634aaf91dbd4cc5b6565e3a493bc82f14da5be not found: ID does not exist" Jul 27 16:20:25.821874 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.821854 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d5c4dfbcd-8h8ft"] Jul 27 16:20:25.822185 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.822167 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 27 16:20:25.848520 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.848490 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf37d2a1-2ccb-49e7-9a33-80b61390d132" path="/var/lib/kubelet/pods/bf37d2a1-2ccb-49e7-9a33-80b61390d132/volumes" Jul 27 16:20:25.883721 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.883696 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-console-config\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.883803 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.883738 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59a1a815-2741-4e8e-a471-e9ebd3556a16-console-serving-cert\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.883803 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.883760 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-oauth-serving-cert\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.883877 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.883827 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-trusted-ca-bundle\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.883877 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.883866 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-service-ca\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.883969 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.883893 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fps5b\" (UniqueName: \"kubernetes.io/projected/59a1a815-2741-4e8e-a471-e9ebd3556a16-kube-api-access-fps5b\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.883969 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.883946 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59a1a815-2741-4e8e-a471-e9ebd3556a16-console-oauth-config\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.909468 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.909434 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 16:20:25.913252 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.913236 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:25.920084 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.920061 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 27 16:20:25.920442 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.920090 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 27 16:20:25.920442 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.920149 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 27 16:20:25.920442 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.920068 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 27 16:20:25.920442 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.920302 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 27 16:20:25.920442 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.920379 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 27 16:20:25.920442 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.920414 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ln7ps\"" Jul 27 16:20:25.920959 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.920943 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 27 16:20:25.921024 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.920946 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 27 16:20:25.925822 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.925806 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 27 16:20:25.930141 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.930122 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 16:20:25.984896 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.984865 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-trusted-ca-bundle\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.984896 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.984910 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15eefb0f-0903-456a-a77e-5608f909512b-config-out\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:25.985093 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.984937 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:25.985093 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.984971 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-service-ca\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.985093 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.985021 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fps5b\" (UniqueName: \"kubernetes.io/projected/59a1a815-2741-4e8e-a471-e9ebd3556a16-kube-api-access-fps5b\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.985093 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.985044 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-web-config\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:25.985093 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.985064 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15eefb0f-0903-456a-a77e-5608f909512b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:25.985093 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.985084 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59a1a815-2741-4e8e-a471-e9ebd3556a16-console-oauth-config\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.985304 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.985122 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:25.985304 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.985158 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b55hg\" (UniqueName: \"kubernetes.io/projected/15eefb0f-0903-456a-a77e-5608f909512b-kube-api-access-b55hg\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:25.985304 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.985190 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/15eefb0f-0903-456a-a77e-5608f909512b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:25.985304 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.985223 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:25.985304 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.985269 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-console-config\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.985304 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.985292 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-config-volume\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:25.985601 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.985391 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15eefb0f-0903-456a-a77e-5608f909512b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:25.985601 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.985428 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15eefb0f-0903-456a-a77e-5608f909512b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:25.985601 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.985452 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:25.985601 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.985494 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:25.985601 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.985573 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59a1a815-2741-4e8e-a471-e9ebd3556a16-console-serving-cert\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.985845 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.985609 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-oauth-serving-cert\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.985845 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.985624 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-service-ca\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.985845 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.985747 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-trusted-ca-bundle\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.986002 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.985933 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-console-config\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.986193 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.986178 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-oauth-serving-cert\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.987516 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.987490 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59a1a815-2741-4e8e-a471-e9ebd3556a16-console-oauth-config\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.987896 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.987880 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59a1a815-2741-4e8e-a471-e9ebd3556a16-console-serving-cert\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:25.994794 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:25.994778 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fps5b\" (UniqueName: \"kubernetes.io/projected/59a1a815-2741-4e8e-a471-e9ebd3556a16-kube-api-access-fps5b\") pod \"console-5d5c4dfbcd-8h8ft\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:26.086057 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.086014 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-web-config\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.086057 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.086050 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15eefb0f-0903-456a-a77e-5608f909512b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.086304 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.086082 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.086304 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.086104 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b55hg\" (UniqueName: \"kubernetes.io/projected/15eefb0f-0903-456a-a77e-5608f909512b-kube-api-access-b55hg\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.086304 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.086122 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/15eefb0f-0903-456a-a77e-5608f909512b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.086748 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.086248 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.086993 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.086769 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-config-volume\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.086993 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.086812 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15eefb0f-0903-456a-a77e-5608f909512b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.086993 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.086847 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15eefb0f-0903-456a-a77e-5608f909512b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.086993 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.086880 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.086993 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.086941 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.087436 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.087032 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15eefb0f-0903-456a-a77e-5608f909512b-config-out\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.087436 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.087067 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.087926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.087708 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/15eefb0f-0903-456a-a77e-5608f909512b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.087926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.087791 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15eefb0f-0903-456a-a77e-5608f909512b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.088971 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.088897 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15eefb0f-0903-456a-a77e-5608f909512b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.089749 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.089721 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.090333 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.090285 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.095441 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.090734 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-config-volume\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.095441 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.092678 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15eefb0f-0903-456a-a77e-5608f909512b-config-out\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.095441 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.093708 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15eefb0f-0903-456a-a77e-5608f909512b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.095441 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.094479 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.095441 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.095346 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.095441 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.095353 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.096641 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.096623 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15eefb0f-0903-456a-a77e-5608f909512b-web-config\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.098312 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.098293 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b55hg\" (UniqueName: \"kubernetes.io/projected/15eefb0f-0903-456a-a77e-5608f909512b-kube-api-access-b55hg\") pod \"alertmanager-main-0\" (UID: \"15eefb0f-0903-456a-a77e-5608f909512b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.118147 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.118126 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:26.223711 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.223678 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 16:20:26.249760 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.249740 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d5c4dfbcd-8h8ft"] Jul 27 16:20:26.252457 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:20:26.252431 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a1a815_2741_4e8e_a471_e9ebd3556a16.slice/crio-031acdddfbf7a528346a2276e43249ed20b64cdc45296189062a6537c7a490f0 WatchSource:0}: Error finding container 031acdddfbf7a528346a2276e43249ed20b64cdc45296189062a6537c7a490f0: Status 404 returned error can't find the container with id 031acdddfbf7a528346a2276e43249ed20b64cdc45296189062a6537c7a490f0 Jul 27 16:20:26.397195 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.397115 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 16:20:26.400194 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:20:26.400170 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15eefb0f_0903_456a_a77e_5608f909512b.slice/crio-80867a632645ddcd88d919e94ccad50a0aafb7ed4c8fe1e50670d0db3a4e6c5a WatchSource:0}: Error finding container 80867a632645ddcd88d919e94ccad50a0aafb7ed4c8fe1e50670d0db3a4e6c5a: Status 404 returned error can't find the container with id 80867a632645ddcd88d919e94ccad50a0aafb7ed4c8fe1e50670d0db3a4e6c5a Jul 27 16:20:26.765942 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.765887 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5c4dfbcd-8h8ft" event={"ID":"59a1a815-2741-4e8e-a471-e9ebd3556a16","Type":"ContainerStarted","Data":"826839c39a2a46f001611f41e16b2ec836007e1794d424218bba83d4bdc8a4d2"} Jul 27 16:20:26.766383 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.765962 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5c4dfbcd-8h8ft" event={"ID":"59a1a815-2741-4e8e-a471-e9ebd3556a16","Type":"ContainerStarted","Data":"031acdddfbf7a528346a2276e43249ed20b64cdc45296189062a6537c7a490f0"} Jul 27 16:20:26.767450 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.767427 2550 generic.go:356] "Generic (PLEG): container finished" podID="15eefb0f-0903-456a-a77e-5608f909512b" containerID="92012ee7d495c8a933d9ea65237ae487dc35acb2b9efa49e06b80cebd79b328b" exitCode=0 Jul 27 16:20:26.767535 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.767522 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15eefb0f-0903-456a-a77e-5608f909512b","Type":"ContainerDied","Data":"92012ee7d495c8a933d9ea65237ae487dc35acb2b9efa49e06b80cebd79b328b"} Jul 27 16:20:26.767591 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.767574 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15eefb0f-0903-456a-a77e-5608f909512b","Type":"ContainerStarted","Data":"80867a632645ddcd88d919e94ccad50a0aafb7ed4c8fe1e50670d0db3a4e6c5a"} Jul 27 16:20:26.788450 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:26.788402 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d5c4dfbcd-8h8ft" podStartSLOduration=1.7883832659999999 podStartE2EDuration="1.788383266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:20:26.787261905 +0000 UTC m=+205.459796405" watchObservedRunningTime="2026-07-27 16:20:26.788383266 +0000 UTC m=+205.460917765" Jul 27 16:20:27.776459 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:27.776423 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15eefb0f-0903-456a-a77e-5608f909512b","Type":"ContainerStarted","Data":"a2fadfd21917625919b2702b37ca9c809a66b07de4fc555457f8edc540487a91"} Jul 27 16:20:27.776949 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:27.776501 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15eefb0f-0903-456a-a77e-5608f909512b","Type":"ContainerStarted","Data":"d4ee87787134144ae2d00c06738a07db00db23da26cb429c4040d7769a2f6713"} Jul 27 16:20:27.776949 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:27.776543 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15eefb0f-0903-456a-a77e-5608f909512b","Type":"ContainerStarted","Data":"02ffda94cc168c092fc0abf5ee155ecbe8470f7d3ba4abbc98a3fb41e7450ee0"} Jul 27 16:20:27.776949 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:27.776583 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15eefb0f-0903-456a-a77e-5608f909512b","Type":"ContainerStarted","Data":"72a52598f5ad701e41db38caea15a632ee896f8b63614cd48f79a9da26e3dbc4"} Jul 27 16:20:27.776949 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:27.776626 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15eefb0f-0903-456a-a77e-5608f909512b","Type":"ContainerStarted","Data":"af24c2e2a16059026b89286b8258caaa26cbcd042cc77997121a2f72f8e7b00a"} Jul 27 16:20:27.776949 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:27.776660 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15eefb0f-0903-456a-a77e-5608f909512b","Type":"ContainerStarted","Data":"b0ce8a280108c7c438884e2af0f407d27148fdebdcf2b3d9e4bfa8d94ad26405"} Jul 27 16:20:27.809635 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:27.809592 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.795342229 podStartE2EDuration="2.809577173s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.234944ms" isStatefulPod=false podCreationTimestamp="2026-07-27 16:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:20:27.805136809 +0000 UTC m=+206.477671307" watchObservedRunningTime="2026-07-27 16:20:27.809577173 +0000 UTC m=+206.482111670" Jul 27 16:20:36.118719 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:36.118682 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:36.119319 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:36.118753 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:36.123394 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:36.123373 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:20:36.819121 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:20:36.819095 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:21:46.981257 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:46.981220 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc"] Jul 27 16:21:46.983786 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:46.983770 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc" Jul 27 16:21:46.986725 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:46.986702 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 16:21:46.986844 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:46.986772 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4gwrm\"" Jul 27 16:21:46.987971 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:46.987958 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 16:21:46.998848 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:46.998820 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc"] Jul 27 16:21:47.062233 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:47.062198 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6070290e-8eea-4ccc-9b89-d4d766cbb231-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc\" (UID: \"6070290e-8eea-4ccc-9b89-d4d766cbb231\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc" Jul 27 16:21:47.062233 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:47.062238 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lhtt\" (UniqueName: \"kubernetes.io/projected/6070290e-8eea-4ccc-9b89-d4d766cbb231-kube-api-access-9lhtt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc\" (UID: \"6070290e-8eea-4ccc-9b89-d4d766cbb231\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc" Jul 27 16:21:47.062445 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:47.062299 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6070290e-8eea-4ccc-9b89-d4d766cbb231-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc\" (UID: \"6070290e-8eea-4ccc-9b89-d4d766cbb231\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc" Jul 27 16:21:47.163042 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:47.163009 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6070290e-8eea-4ccc-9b89-d4d766cbb231-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc\" (UID: \"6070290e-8eea-4ccc-9b89-d4d766cbb231\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc" Jul 27 16:21:47.163233 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:47.163102 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6070290e-8eea-4ccc-9b89-d4d766cbb231-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc\" (UID: \"6070290e-8eea-4ccc-9b89-d4d766cbb231\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc" Jul 27 16:21:47.163233 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:47.163135 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lhtt\" (UniqueName: \"kubernetes.io/projected/6070290e-8eea-4ccc-9b89-d4d766cbb231-kube-api-access-9lhtt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc\" (UID: \"6070290e-8eea-4ccc-9b89-d4d766cbb231\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc" Jul 27 16:21:47.163431 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:47.163411 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6070290e-8eea-4ccc-9b89-d4d766cbb231-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc\" (UID: \"6070290e-8eea-4ccc-9b89-d4d766cbb231\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc" Jul 27 16:21:47.163502 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:47.163431 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6070290e-8eea-4ccc-9b89-d4d766cbb231-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc\" (UID: \"6070290e-8eea-4ccc-9b89-d4d766cbb231\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc" Jul 27 16:21:47.178494 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:47.178468 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lhtt\" (UniqueName: \"kubernetes.io/projected/6070290e-8eea-4ccc-9b89-d4d766cbb231-kube-api-access-9lhtt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc\" (UID: \"6070290e-8eea-4ccc-9b89-d4d766cbb231\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc" Jul 27 16:21:47.292715 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:47.292685 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc" Jul 27 16:21:47.453584 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:47.453550 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc"] Jul 27 16:21:47.458430 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:21:47.458402 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6070290e_8eea_4ccc_9b89_d4d766cbb231.slice/crio-52040d27fee0c61cd9b4ce28d275e7a90890cca6089f97dc86edda065cb98772 WatchSource:0}: Error finding container 52040d27fee0c61cd9b4ce28d275e7a90890cca6089f97dc86edda065cb98772: Status 404 returned error can't find the container with id 52040d27fee0c61cd9b4ce28d275e7a90890cca6089f97dc86edda065cb98772 Jul 27 16:21:48.108976 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:48.108941 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc" event={"ID":"6070290e-8eea-4ccc-9b89-d4d766cbb231","Type":"ContainerStarted","Data":"52040d27fee0c61cd9b4ce28d275e7a90890cca6089f97dc86edda065cb98772"} Jul 27 16:21:55.143373 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:55.143339 2550 generic.go:356] "Generic (PLEG): container finished" podID="6070290e-8eea-4ccc-9b89-d4d766cbb231" containerID="08b72fc1e4eef07db86830bb362dacd9e44be1f205fd2d7b93172b37fc6f477e" exitCode=0 Jul 27 16:21:55.143762 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:55.143379 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc" event={"ID":"6070290e-8eea-4ccc-9b89-d4d766cbb231","Type":"ContainerDied","Data":"08b72fc1e4eef07db86830bb362dacd9e44be1f205fd2d7b93172b37fc6f477e"} Jul 27 16:21:58.158996 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:58.158958 2550 generic.go:356] "Generic (PLEG): container finished" podID="6070290e-8eea-4ccc-9b89-d4d766cbb231" containerID="fd5be280c1d0cbb6091684c56587caeff696ffefe75ed52b3004379a752dd756" exitCode=0 Jul 27 16:21:58.159363 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:21:58.159049 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc" event={"ID":"6070290e-8eea-4ccc-9b89-d4d766cbb231","Type":"ContainerDied","Data":"fd5be280c1d0cbb6091684c56587caeff696ffefe75ed52b3004379a752dd756"} Jul 27 16:22:01.718330 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:01.718300 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:22:01.725801 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:01.725779 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:22:01.727412 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:01.727393 2550 kubelet.go:1648] "Image garbage collection succeeded" Jul 27 16:22:07.203596 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:07.203563 2550 generic.go:356] "Generic (PLEG): container finished" podID="6070290e-8eea-4ccc-9b89-d4d766cbb231" containerID="4c065949d415531855ebf88c276f718703b37667e4535d0f540167b76266873e" exitCode=0 Jul 27 16:22:07.204073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:07.203618 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc" event={"ID":"6070290e-8eea-4ccc-9b89-d4d766cbb231","Type":"ContainerDied","Data":"4c065949d415531855ebf88c276f718703b37667e4535d0f540167b76266873e"} Jul 27 16:22:08.351100 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:08.351075 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc" Jul 27 16:22:08.448358 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:08.448328 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6070290e-8eea-4ccc-9b89-d4d766cbb231-bundle\") pod \"6070290e-8eea-4ccc-9b89-d4d766cbb231\" (UID: \"6070290e-8eea-4ccc-9b89-d4d766cbb231\") " Jul 27 16:22:08.448530 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:08.448378 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lhtt\" (UniqueName: \"kubernetes.io/projected/6070290e-8eea-4ccc-9b89-d4d766cbb231-kube-api-access-9lhtt\") pod \"6070290e-8eea-4ccc-9b89-d4d766cbb231\" (UID: \"6070290e-8eea-4ccc-9b89-d4d766cbb231\") " Jul 27 16:22:08.448530 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:08.448414 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6070290e-8eea-4ccc-9b89-d4d766cbb231-util\") pod \"6070290e-8eea-4ccc-9b89-d4d766cbb231\" (UID: \"6070290e-8eea-4ccc-9b89-d4d766cbb231\") " Jul 27 16:22:08.449049 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:08.449006 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6070290e-8eea-4ccc-9b89-d4d766cbb231-bundle" (OuterVolumeSpecName: "bundle") pod "6070290e-8eea-4ccc-9b89-d4d766cbb231" (UID: "6070290e-8eea-4ccc-9b89-d4d766cbb231"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:22:08.450690 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:08.450669 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6070290e-8eea-4ccc-9b89-d4d766cbb231-kube-api-access-9lhtt" (OuterVolumeSpecName: "kube-api-access-9lhtt") pod "6070290e-8eea-4ccc-9b89-d4d766cbb231" (UID: "6070290e-8eea-4ccc-9b89-d4d766cbb231"). InnerVolumeSpecName "kube-api-access-9lhtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:22:08.454876 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:08.454849 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6070290e-8eea-4ccc-9b89-d4d766cbb231-util" (OuterVolumeSpecName: "util") pod "6070290e-8eea-4ccc-9b89-d4d766cbb231" (UID: "6070290e-8eea-4ccc-9b89-d4d766cbb231"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:22:08.549612 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:08.549581 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6070290e-8eea-4ccc-9b89-d4d766cbb231-bundle\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:22:08.549612 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:08.549606 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lhtt\" (UniqueName: \"kubernetes.io/projected/6070290e-8eea-4ccc-9b89-d4d766cbb231-kube-api-access-9lhtt\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:22:08.549612 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:08.549616 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6070290e-8eea-4ccc-9b89-d4d766cbb231-util\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:22:09.213634 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:09.213603 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc" Jul 27 16:22:09.213634 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:09.213617 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ks9zc" event={"ID":"6070290e-8eea-4ccc-9b89-d4d766cbb231","Type":"ContainerDied","Data":"52040d27fee0c61cd9b4ce28d275e7a90890cca6089f97dc86edda065cb98772"} Jul 27 16:22:09.213832 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:09.213657 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52040d27fee0c61cd9b4ce28d275e7a90890cca6089f97dc86edda065cb98772" Jul 27 16:22:15.550035 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.549997 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ft7xj"] Jul 27 16:22:15.550519 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.550351 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6070290e-8eea-4ccc-9b89-d4d766cbb231" containerName="extract" Jul 27 16:22:15.550519 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.550363 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6070290e-8eea-4ccc-9b89-d4d766cbb231" containerName="extract" Jul 27 16:22:15.550519 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.550379 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6070290e-8eea-4ccc-9b89-d4d766cbb231" containerName="util" Jul 27 16:22:15.550519 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.550385 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6070290e-8eea-4ccc-9b89-d4d766cbb231" containerName="util" Jul 27 16:22:15.550519 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.550398 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6070290e-8eea-4ccc-9b89-d4d766cbb231" containerName="pull" Jul 27 16:22:15.550519 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.550404 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6070290e-8eea-4ccc-9b89-d4d766cbb231" containerName="pull" Jul 27 16:22:15.550519 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.550462 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6070290e-8eea-4ccc-9b89-d4d766cbb231" containerName="extract" Jul 27 16:22:15.553979 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.553963 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ft7xj" Jul 27 16:22:15.558017 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.557981 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 27 16:22:15.558125 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.558030 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-72jj9\"" Jul 27 16:22:15.558125 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.558089 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 27 16:22:15.576182 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.576156 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ft7xj"] Jul 27 16:22:15.705914 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.705877 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk6j7\" (UniqueName: \"kubernetes.io/projected/2d7cc3eb-d89a-4307-bf7a-488b907f6c7f-kube-api-access-sk6j7\") pod \"cert-manager-operator-controller-manager-55c444bf86-ft7xj\" (UID: \"2d7cc3eb-d89a-4307-bf7a-488b907f6c7f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ft7xj" Jul 27 16:22:15.706089 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.705936 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d7cc3eb-d89a-4307-bf7a-488b907f6c7f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ft7xj\" (UID: \"2d7cc3eb-d89a-4307-bf7a-488b907f6c7f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ft7xj" Jul 27 16:22:15.807283 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.807207 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk6j7\" (UniqueName: \"kubernetes.io/projected/2d7cc3eb-d89a-4307-bf7a-488b907f6c7f-kube-api-access-sk6j7\") pod \"cert-manager-operator-controller-manager-55c444bf86-ft7xj\" (UID: \"2d7cc3eb-d89a-4307-bf7a-488b907f6c7f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ft7xj" Jul 27 16:22:15.807283 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.807242 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d7cc3eb-d89a-4307-bf7a-488b907f6c7f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ft7xj\" (UID: \"2d7cc3eb-d89a-4307-bf7a-488b907f6c7f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ft7xj" Jul 27 16:22:15.807591 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.807577 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d7cc3eb-d89a-4307-bf7a-488b907f6c7f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ft7xj\" (UID: \"2d7cc3eb-d89a-4307-bf7a-488b907f6c7f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ft7xj" Jul 27 16:22:15.817106 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.817084 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk6j7\" (UniqueName: \"kubernetes.io/projected/2d7cc3eb-d89a-4307-bf7a-488b907f6c7f-kube-api-access-sk6j7\") pod \"cert-manager-operator-controller-manager-55c444bf86-ft7xj\" (UID: \"2d7cc3eb-d89a-4307-bf7a-488b907f6c7f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ft7xj" Jul 27 16:22:15.862745 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:15.862719 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ft7xj" Jul 27 16:22:16.014012 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:16.013982 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ft7xj"] Jul 27 16:22:16.018609 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:22:16.018582 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7cc3eb_d89a_4307_bf7a_488b907f6c7f.slice/crio-d35b39914e963d6ddf817f0c5df137a8274d1f657b4bc07ce4189394d7d413bc WatchSource:0}: Error finding container d35b39914e963d6ddf817f0c5df137a8274d1f657b4bc07ce4189394d7d413bc: Status 404 returned error can't find the container with id d35b39914e963d6ddf817f0c5df137a8274d1f657b4bc07ce4189394d7d413bc Jul 27 16:22:16.021282 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:16.021264 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 16:22:16.244529 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:16.244453 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ft7xj" event={"ID":"2d7cc3eb-d89a-4307-bf7a-488b907f6c7f","Type":"ContainerStarted","Data":"d35b39914e963d6ddf817f0c5df137a8274d1f657b4bc07ce4189394d7d413bc"} Jul 27 16:22:18.255922 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:18.255867 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ft7xj" event={"ID":"2d7cc3eb-d89a-4307-bf7a-488b907f6c7f","Type":"ContainerStarted","Data":"77a0a36b1a660a548bf48f280d445d978057c6eb964ac7abdcd8ffb191b1f3a4"} Jul 27 16:22:18.284736 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:18.284688 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ft7xj" podStartSLOduration=1.644173398 podStartE2EDuration="3.284673123s" totalImagesPullingTime="1.640499725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:22:18.283565482 +0000 UTC m=+316.956099981" watchObservedRunningTime="2026-07-27 16:22:18.284673123 +0000 UTC m=+316.957207621" Jul 27 16:22:18.978829 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:18.978794 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb"] Jul 27 16:22:18.981374 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:18.981359 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb" Jul 27 16:22:18.984552 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:18.984530 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4gwrm\"" Jul 27 16:22:18.984890 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:18.984877 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 16:22:18.986954 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:18.986931 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 16:22:19.011804 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:19.011779 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb"] Jul 27 16:22:19.138354 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:19.138323 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd4d2d4e-66d6-4981-bafc-cd622c1a84fb-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb\" (UID: \"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb" Jul 27 16:22:19.138354 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:19.138357 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd4d2d4e-66d6-4981-bafc-cd622c1a84fb-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb\" (UID: \"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb" Jul 27 16:22:19.138558 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:19.138383 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9qtc\" (UniqueName: \"kubernetes.io/projected/dd4d2d4e-66d6-4981-bafc-cd622c1a84fb-kube-api-access-z9qtc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb\" (UID: \"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb" Jul 27 16:22:19.239861 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:19.239781 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9qtc\" (UniqueName: \"kubernetes.io/projected/dd4d2d4e-66d6-4981-bafc-cd622c1a84fb-kube-api-access-z9qtc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb\" (UID: \"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb" Jul 27 16:22:19.240029 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:19.239877 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd4d2d4e-66d6-4981-bafc-cd622c1a84fb-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb\" (UID: \"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb" Jul 27 16:22:19.240029 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:19.239926 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd4d2d4e-66d6-4981-bafc-cd622c1a84fb-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb\" (UID: \"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb" Jul 27 16:22:19.240301 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:19.240286 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd4d2d4e-66d6-4981-bafc-cd622c1a84fb-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb\" (UID: \"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb" Jul 27 16:22:19.240338 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:19.240304 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd4d2d4e-66d6-4981-bafc-cd622c1a84fb-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb\" (UID: \"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb" Jul 27 16:22:19.259976 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:19.259943 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9qtc\" (UniqueName: \"kubernetes.io/projected/dd4d2d4e-66d6-4981-bafc-cd622c1a84fb-kube-api-access-z9qtc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb\" (UID: \"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb" Jul 27 16:22:19.290315 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:19.290288 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb" Jul 27 16:22:19.461108 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:19.461078 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb"] Jul 27 16:22:19.464675 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:22:19.464636 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4d2d4e_66d6_4981_bafc_cd622c1a84fb.slice/crio-094500d3b9438eaa32481cd3ef6f8b8ecb66277b61ee27597a0de90f58a94e36 WatchSource:0}: Error finding container 094500d3b9438eaa32481cd3ef6f8b8ecb66277b61ee27597a0de90f58a94e36: Status 404 returned error can't find the container with id 094500d3b9438eaa32481cd3ef6f8b8ecb66277b61ee27597a0de90f58a94e36 Jul 27 16:22:20.267395 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:20.267365 2550 generic.go:356] "Generic (PLEG): container finished" podID="dd4d2d4e-66d6-4981-bafc-cd622c1a84fb" containerID="2a8bddc09306c68eb6c5648d094c6860d7b90dbba08ccf41dea9516e9097db24" exitCode=0 Jul 27 16:22:20.267757 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:20.267440 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb" event={"ID":"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb","Type":"ContainerDied","Data":"2a8bddc09306c68eb6c5648d094c6860d7b90dbba08ccf41dea9516e9097db24"} Jul 27 16:22:20.267757 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:20.267481 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb" event={"ID":"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb","Type":"ContainerStarted","Data":"094500d3b9438eaa32481cd3ef6f8b8ecb66277b61ee27597a0de90f58a94e36"} Jul 27 16:22:23.023729 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:23.023696 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-89sn6"] Jul 27 16:22:23.026182 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:23.026161 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89sn6" Jul 27 16:22:23.032523 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:23.032498 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 27 16:22:23.032716 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:23.032559 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-k4gcm\"" Jul 27 16:22:23.035191 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:23.035163 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 27 16:22:23.045716 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:23.045687 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-89sn6"] Jul 27 16:22:23.174012 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:23.173988 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd4k9\" (UniqueName: \"kubernetes.io/projected/ec3d7d99-7ef6-4dc6-852e-5c2f236d55bc-kube-api-access-dd4k9\") pod \"cert-manager-cainjector-559cdd8c87-89sn6\" (UID: \"ec3d7d99-7ef6-4dc6-852e-5c2f236d55bc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89sn6" Jul 27 16:22:23.174118 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:23.174027 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec3d7d99-7ef6-4dc6-852e-5c2f236d55bc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-89sn6\" (UID: \"ec3d7d99-7ef6-4dc6-852e-5c2f236d55bc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89sn6" Jul 27 16:22:23.274745 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:23.274679 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd4k9\" (UniqueName: \"kubernetes.io/projected/ec3d7d99-7ef6-4dc6-852e-5c2f236d55bc-kube-api-access-dd4k9\") pod \"cert-manager-cainjector-559cdd8c87-89sn6\" (UID: \"ec3d7d99-7ef6-4dc6-852e-5c2f236d55bc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89sn6" Jul 27 16:22:23.274745 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:23.274719 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec3d7d99-7ef6-4dc6-852e-5c2f236d55bc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-89sn6\" (UID: \"ec3d7d99-7ef6-4dc6-852e-5c2f236d55bc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89sn6" Jul 27 16:22:23.283541 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:23.283507 2550 generic.go:356] "Generic (PLEG): container finished" podID="dd4d2d4e-66d6-4981-bafc-cd622c1a84fb" containerID="708db74fae21236acc9a584846367de0a851f3381a01affd7046fcb71726a98b" exitCode=0 Jul 27 16:22:23.283667 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:23.283584 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb" event={"ID":"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb","Type":"ContainerDied","Data":"708db74fae21236acc9a584846367de0a851f3381a01affd7046fcb71726a98b"} Jul 27 16:22:23.286633 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:23.286616 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd4k9\" (UniqueName: \"kubernetes.io/projected/ec3d7d99-7ef6-4dc6-852e-5c2f236d55bc-kube-api-access-dd4k9\") pod \"cert-manager-cainjector-559cdd8c87-89sn6\" (UID: \"ec3d7d99-7ef6-4dc6-852e-5c2f236d55bc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89sn6" Jul 27 16:22:23.287583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:23.287564 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec3d7d99-7ef6-4dc6-852e-5c2f236d55bc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-89sn6\" (UID: \"ec3d7d99-7ef6-4dc6-852e-5c2f236d55bc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89sn6" Jul 27 16:22:23.344885 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:23.344862 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89sn6" Jul 27 16:22:23.483258 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:23.483235 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-89sn6"] Jul 27 16:22:23.485405 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:22:23.485383 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec3d7d99_7ef6_4dc6_852e_5c2f236d55bc.slice/crio-ef3f0bff4cdf5a21ea58402a3eede3b2952032a09f3d61e03852301f82020858 WatchSource:0}: Error finding container ef3f0bff4cdf5a21ea58402a3eede3b2952032a09f3d61e03852301f82020858: Status 404 returned error can't find the container with id ef3f0bff4cdf5a21ea58402a3eede3b2952032a09f3d61e03852301f82020858 Jul 27 16:22:24.289452 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:24.289417 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89sn6" event={"ID":"ec3d7d99-7ef6-4dc6-852e-5c2f236d55bc","Type":"ContainerStarted","Data":"ef3f0bff4cdf5a21ea58402a3eede3b2952032a09f3d61e03852301f82020858"} Jul 27 16:22:24.291410 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:24.291390 2550 generic.go:356] "Generic (PLEG): container finished" podID="dd4d2d4e-66d6-4981-bafc-cd622c1a84fb" containerID="dfb32ee92750f2229de4bebed81e97994101f3d896db1766482adcb5f7a01d54" exitCode=0 Jul 27 16:22:24.291483 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:24.291446 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb" event={"ID":"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb","Type":"ContainerDied","Data":"dfb32ee92750f2229de4bebed81e97994101f3d896db1766482adcb5f7a01d54"} Jul 27 16:22:26.285313 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:26.285289 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb" Jul 27 16:22:26.302671 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:26.302643 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb" event={"ID":"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb","Type":"ContainerDied","Data":"094500d3b9438eaa32481cd3ef6f8b8ecb66277b61ee27597a0de90f58a94e36"} Jul 27 16:22:26.302671 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:26.302671 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzvnvb" Jul 27 16:22:26.302833 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:26.302686 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="094500d3b9438eaa32481cd3ef6f8b8ecb66277b61ee27597a0de90f58a94e36" Jul 27 16:22:26.402460 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:26.402432 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9qtc\" (UniqueName: \"kubernetes.io/projected/dd4d2d4e-66d6-4981-bafc-cd622c1a84fb-kube-api-access-z9qtc\") pod \"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb\" (UID: \"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb\") " Jul 27 16:22:26.402609 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:26.402488 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd4d2d4e-66d6-4981-bafc-cd622c1a84fb-util\") pod \"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb\" (UID: \"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb\") " Jul 27 16:22:26.402609 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:26.402508 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd4d2d4e-66d6-4981-bafc-cd622c1a84fb-bundle\") pod \"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb\" (UID: \"dd4d2d4e-66d6-4981-bafc-cd622c1a84fb\") " Jul 27 16:22:26.402890 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:26.402865 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4d2d4e-66d6-4981-bafc-cd622c1a84fb-bundle" (OuterVolumeSpecName: "bundle") pod "dd4d2d4e-66d6-4981-bafc-cd622c1a84fb" (UID: "dd4d2d4e-66d6-4981-bafc-cd622c1a84fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:22:26.404879 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:26.404853 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4d2d4e-66d6-4981-bafc-cd622c1a84fb-kube-api-access-z9qtc" (OuterVolumeSpecName: "kube-api-access-z9qtc") pod "dd4d2d4e-66d6-4981-bafc-cd622c1a84fb" (UID: "dd4d2d4e-66d6-4981-bafc-cd622c1a84fb"). InnerVolumeSpecName "kube-api-access-z9qtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:22:26.408766 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:26.408730 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4d2d4e-66d6-4981-bafc-cd622c1a84fb-util" (OuterVolumeSpecName: "util") pod "dd4d2d4e-66d6-4981-bafc-cd622c1a84fb" (UID: "dd4d2d4e-66d6-4981-bafc-cd622c1a84fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:22:26.503717 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:26.503673 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd4d2d4e-66d6-4981-bafc-cd622c1a84fb-util\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:22:26.503717 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:26.503704 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd4d2d4e-66d6-4981-bafc-cd622c1a84fb-bundle\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:22:26.503717 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:26.503718 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9qtc\" (UniqueName: \"kubernetes.io/projected/dd4d2d4e-66d6-4981-bafc-cd622c1a84fb-kube-api-access-z9qtc\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:22:27.308608 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:27.308569 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89sn6" event={"ID":"ec3d7d99-7ef6-4dc6-852e-5c2f236d55bc","Type":"ContainerStarted","Data":"e2c536acdba75cfec9b4db312875ac2fb059e01006b1c4a237764588b07e1f03"} Jul 27 16:22:27.341759 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:27.341700 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89sn6" podStartSLOduration=2.52430906 podStartE2EDuration="5.341681471s" totalImagesPullingTime="2.817372411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:22:27.339709854 +0000 UTC m=+326.012244357" watchObservedRunningTime="2026-07-27 16:22:27.341681471 +0000 UTC m=+326.014215970" Jul 27 16:22:31.977963 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:31.977927 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4xtt2"] Jul 27 16:22:31.978317 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:31.978260 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd4d2d4e-66d6-4981-bafc-cd622c1a84fb" containerName="extract" Jul 27 16:22:31.978317 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:31.978272 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd4d2d4e-66d6-4981-bafc-cd622c1a84fb" containerName="extract" Jul 27 16:22:31.978317 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:31.978282 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd4d2d4e-66d6-4981-bafc-cd622c1a84fb" containerName="pull" Jul 27 16:22:31.978317 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:31.978288 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd4d2d4e-66d6-4981-bafc-cd622c1a84fb" containerName="pull" Jul 27 16:22:31.978317 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:31.978306 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd4d2d4e-66d6-4981-bafc-cd622c1a84fb" containerName="util" Jul 27 16:22:31.978317 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:31.978311 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd4d2d4e-66d6-4981-bafc-cd622c1a84fb" containerName="util" Jul 27 16:22:31.978516 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:31.978365 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd4d2d4e-66d6-4981-bafc-cd622c1a84fb" containerName="extract" Jul 27 16:22:31.982099 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:31.982081 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4xtt2" Jul 27 16:22:31.985962 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:31.985940 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-pq9v6\"" Jul 27 16:22:31.996625 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:31.996599 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4xtt2"] Jul 27 16:22:32.151417 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:32.151385 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4ca4cee-3b3e-4183-a276-0be7e8642e6b-bound-sa-token\") pod \"cert-manager-55b4d954d5-4xtt2\" (UID: \"a4ca4cee-3b3e-4183-a276-0be7e8642e6b\") " pod="cert-manager/cert-manager-55b4d954d5-4xtt2" Jul 27 16:22:32.151578 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:32.151428 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr2v5\" (UniqueName: \"kubernetes.io/projected/a4ca4cee-3b3e-4183-a276-0be7e8642e6b-kube-api-access-tr2v5\") pod \"cert-manager-55b4d954d5-4xtt2\" (UID: \"a4ca4cee-3b3e-4183-a276-0be7e8642e6b\") " pod="cert-manager/cert-manager-55b4d954d5-4xtt2" Jul 27 16:22:32.252512 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:32.252428 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4ca4cee-3b3e-4183-a276-0be7e8642e6b-bound-sa-token\") pod \"cert-manager-55b4d954d5-4xtt2\" (UID: \"a4ca4cee-3b3e-4183-a276-0be7e8642e6b\") " pod="cert-manager/cert-manager-55b4d954d5-4xtt2" Jul 27 16:22:32.252512 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:32.252466 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr2v5\" (UniqueName: \"kubernetes.io/projected/a4ca4cee-3b3e-4183-a276-0be7e8642e6b-kube-api-access-tr2v5\") pod \"cert-manager-55b4d954d5-4xtt2\" (UID: \"a4ca4cee-3b3e-4183-a276-0be7e8642e6b\") " pod="cert-manager/cert-manager-55b4d954d5-4xtt2" Jul 27 16:22:32.261365 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:32.261330 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4ca4cee-3b3e-4183-a276-0be7e8642e6b-bound-sa-token\") pod \"cert-manager-55b4d954d5-4xtt2\" (UID: \"a4ca4cee-3b3e-4183-a276-0be7e8642e6b\") " pod="cert-manager/cert-manager-55b4d954d5-4xtt2" Jul 27 16:22:32.261504 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:32.261477 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr2v5\" (UniqueName: \"kubernetes.io/projected/a4ca4cee-3b3e-4183-a276-0be7e8642e6b-kube-api-access-tr2v5\") pod \"cert-manager-55b4d954d5-4xtt2\" (UID: \"a4ca4cee-3b3e-4183-a276-0be7e8642e6b\") " pod="cert-manager/cert-manager-55b4d954d5-4xtt2" Jul 27 16:22:32.295436 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:32.295406 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4xtt2" Jul 27 16:22:32.429149 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:32.429119 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4xtt2"] Jul 27 16:22:32.432640 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:22:32.432618 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4ca4cee_3b3e_4183_a276_0be7e8642e6b.slice/crio-246cf9731be889c16f33b657346b935f4a052cd381c5e7272b841efa3a09567f WatchSource:0}: Error finding container 246cf9731be889c16f33b657346b935f4a052cd381c5e7272b841efa3a09567f: Status 404 returned error can't find the container with id 246cf9731be889c16f33b657346b935f4a052cd381c5e7272b841efa3a09567f Jul 27 16:22:33.338031 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:33.337992 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4xtt2" event={"ID":"a4ca4cee-3b3e-4183-a276-0be7e8642e6b","Type":"ContainerStarted","Data":"7612db413c5597e31cebbed191327104a5d207071529f399402a1ba51cb4a360"} Jul 27 16:22:33.338435 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:33.338040 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4xtt2" event={"ID":"a4ca4cee-3b3e-4183-a276-0be7e8642e6b","Type":"ContainerStarted","Data":"246cf9731be889c16f33b657346b935f4a052cd381c5e7272b841efa3a09567f"} Jul 27 16:22:33.357313 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:33.357260 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-4xtt2" podStartSLOduration=2.357242661 podStartE2EDuration="2.357242661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:22:33.35654464 +0000 UTC m=+332.029079141" watchObservedRunningTime="2026-07-27 16:22:33.357242661 +0000 UTC m=+332.029777160" Jul 27 16:22:43.342912 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:43.342864 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h"] Jul 27 16:22:43.348436 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:43.348420 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h" Jul 27 16:22:43.351223 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:43.351202 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 16:22:43.351357 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:43.351246 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 16:22:43.351357 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:43.351294 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4gwrm\"" Jul 27 16:22:43.355507 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:43.355483 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h"] Jul 27 16:22:43.445191 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:43.445155 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ace6f3f1-a25c-4e9b-ab97-c288f8bae03a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h\" (UID: \"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h" Jul 27 16:22:43.445375 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:43.445205 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ace6f3f1-a25c-4e9b-ab97-c288f8bae03a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h\" (UID: \"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h" Jul 27 16:22:43.445375 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:43.445284 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck44h\" (UniqueName: \"kubernetes.io/projected/ace6f3f1-a25c-4e9b-ab97-c288f8bae03a-kube-api-access-ck44h\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h\" (UID: \"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h" Jul 27 16:22:43.546219 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:43.546174 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck44h\" (UniqueName: \"kubernetes.io/projected/ace6f3f1-a25c-4e9b-ab97-c288f8bae03a-kube-api-access-ck44h\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h\" (UID: \"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h" Jul 27 16:22:43.546418 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:43.546241 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ace6f3f1-a25c-4e9b-ab97-c288f8bae03a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h\" (UID: \"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h" Jul 27 16:22:43.546418 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:43.546274 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ace6f3f1-a25c-4e9b-ab97-c288f8bae03a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h\" (UID: \"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h" Jul 27 16:22:43.546624 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:43.546606 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ace6f3f1-a25c-4e9b-ab97-c288f8bae03a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h\" (UID: \"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h" Jul 27 16:22:43.546682 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:43.546630 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ace6f3f1-a25c-4e9b-ab97-c288f8bae03a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h\" (UID: \"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h" Jul 27 16:22:43.556793 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:43.556770 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck44h\" (UniqueName: \"kubernetes.io/projected/ace6f3f1-a25c-4e9b-ab97-c288f8bae03a-kube-api-access-ck44h\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h\" (UID: \"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h" Jul 27 16:22:43.659208 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:43.659124 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h" Jul 27 16:22:43.794655 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:43.794633 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h"] Jul 27 16:22:43.796842 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:22:43.796814 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace6f3f1_a25c_4e9b_ab97_c288f8bae03a.slice/crio-40b72fdc9bf5a47b746d062d2355dc71d7e36759986ddbcc8fa3b6ec2d333bf9 WatchSource:0}: Error finding container 40b72fdc9bf5a47b746d062d2355dc71d7e36759986ddbcc8fa3b6ec2d333bf9: Status 404 returned error can't find the container with id 40b72fdc9bf5a47b746d062d2355dc71d7e36759986ddbcc8fa3b6ec2d333bf9 Jul 27 16:22:44.391599 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:44.391560 2550 generic.go:356] "Generic (PLEG): container finished" podID="ace6f3f1-a25c-4e9b-ab97-c288f8bae03a" containerID="a387ddd6f93e34738c48fab34eb0b72f116db7e2c16a3551e1300128f3cd0ed3" exitCode=0 Jul 27 16:22:44.391985 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:44.391648 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h" event={"ID":"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a","Type":"ContainerDied","Data":"a387ddd6f93e34738c48fab34eb0b72f116db7e2c16a3551e1300128f3cd0ed3"} Jul 27 16:22:44.391985 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:44.391696 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h" event={"ID":"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a","Type":"ContainerStarted","Data":"40b72fdc9bf5a47b746d062d2355dc71d7e36759986ddbcc8fa3b6ec2d333bf9"} Jul 27 16:22:47.407043 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:47.407013 2550 generic.go:356] "Generic (PLEG): container finished" podID="ace6f3f1-a25c-4e9b-ab97-c288f8bae03a" containerID="f98d70f269ae4e636fc69dc1be53a7485087d4a8ab629c54e4770c0a3f6cec7f" exitCode=0 Jul 27 16:22:47.407435 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:47.407094 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h" event={"ID":"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a","Type":"ContainerDied","Data":"f98d70f269ae4e636fc69dc1be53a7485087d4a8ab629c54e4770c0a3f6cec7f"} Jul 27 16:22:48.413927 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:48.413864 2550 generic.go:356] "Generic (PLEG): container finished" podID="ace6f3f1-a25c-4e9b-ab97-c288f8bae03a" containerID="9688439f3c2b73f3685c53da975a5d7da743b5c096e58c2c497ebd99ee73e361" exitCode=0 Jul 27 16:22:48.414308 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:48.413941 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h" event={"ID":"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a","Type":"ContainerDied","Data":"9688439f3c2b73f3685c53da975a5d7da743b5c096e58c2c497ebd99ee73e361"} Jul 27 16:22:49.545925 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:49.545879 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h" Jul 27 16:22:49.595540 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:49.595504 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck44h\" (UniqueName: \"kubernetes.io/projected/ace6f3f1-a25c-4e9b-ab97-c288f8bae03a-kube-api-access-ck44h\") pod \"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a\" (UID: \"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a\") " Jul 27 16:22:49.595706 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:49.595551 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ace6f3f1-a25c-4e9b-ab97-c288f8bae03a-bundle\") pod \"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a\" (UID: \"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a\") " Jul 27 16:22:49.595706 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:49.595585 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ace6f3f1-a25c-4e9b-ab97-c288f8bae03a-util\") pod \"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a\" (UID: \"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a\") " Jul 27 16:22:49.596457 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:49.596423 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace6f3f1-a25c-4e9b-ab97-c288f8bae03a-bundle" (OuterVolumeSpecName: "bundle") pod "ace6f3f1-a25c-4e9b-ab97-c288f8bae03a" (UID: "ace6f3f1-a25c-4e9b-ab97-c288f8bae03a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:22:49.597788 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:49.597767 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace6f3f1-a25c-4e9b-ab97-c288f8bae03a-kube-api-access-ck44h" (OuterVolumeSpecName: "kube-api-access-ck44h") pod "ace6f3f1-a25c-4e9b-ab97-c288f8bae03a" (UID: "ace6f3f1-a25c-4e9b-ab97-c288f8bae03a"). InnerVolumeSpecName "kube-api-access-ck44h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:22:49.601084 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:49.601063 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace6f3f1-a25c-4e9b-ab97-c288f8bae03a-util" (OuterVolumeSpecName: "util") pod "ace6f3f1-a25c-4e9b-ab97-c288f8bae03a" (UID: "ace6f3f1-a25c-4e9b-ab97-c288f8bae03a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:22:49.696864 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:49.696786 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ace6f3f1-a25c-4e9b-ab97-c288f8bae03a-util\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:22:49.696864 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:49.696812 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck44h\" (UniqueName: \"kubernetes.io/projected/ace6f3f1-a25c-4e9b-ab97-c288f8bae03a-kube-api-access-ck44h\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:22:49.696864 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:49.696822 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ace6f3f1-a25c-4e9b-ab97-c288f8bae03a-bundle\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:22:50.425423 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:50.425390 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h" Jul 27 16:22:50.425423 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:50.425403 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8865h" event={"ID":"ace6f3f1-a25c-4e9b-ab97-c288f8bae03a","Type":"ContainerDied","Data":"40b72fdc9bf5a47b746d062d2355dc71d7e36759986ddbcc8fa3b6ec2d333bf9"} Jul 27 16:22:50.425658 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:50.425454 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40b72fdc9bf5a47b746d062d2355dc71d7e36759986ddbcc8fa3b6ec2d333bf9" Jul 27 16:22:59.214643 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.214567 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp"] Jul 27 16:22:59.215077 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.214918 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ace6f3f1-a25c-4e9b-ab97-c288f8bae03a" containerName="pull" Jul 27 16:22:59.215077 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.214930 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ace6f3f1-a25c-4e9b-ab97-c288f8bae03a" containerName="pull" Jul 27 16:22:59.215077 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.214938 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ace6f3f1-a25c-4e9b-ab97-c288f8bae03a" containerName="extract" Jul 27 16:22:59.215077 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.214944 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ace6f3f1-a25c-4e9b-ab97-c288f8bae03a" containerName="extract" Jul 27 16:22:59.215077 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.214964 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ace6f3f1-a25c-4e9b-ab97-c288f8bae03a" containerName="util" Jul 27 16:22:59.215077 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.214969 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ace6f3f1-a25c-4e9b-ab97-c288f8bae03a" containerName="util" Jul 27 16:22:59.215077 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.215021 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ace6f3f1-a25c-4e9b-ab97-c288f8bae03a" containerName="extract" Jul 27 16:22:59.217252 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.217236 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" Jul 27 16:22:59.220848 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.220830 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 27 16:22:59.221006 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.220891 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 27 16:22:59.221006 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.220983 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 27 16:22:59.221356 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.221340 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 27 16:22:59.221443 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.221365 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jgz5w\"" Jul 27 16:22:59.241404 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.241382 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp"] Jul 27 16:22:59.278827 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.278783 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2730b6a9-a99f-4a61-b69c-74d48ee48c74-webhook-cert\") pod \"opendatahub-operator-controller-manager-55548c94b5-9zdpp\" (UID: \"2730b6a9-a99f-4a61-b69c-74d48ee48c74\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" Jul 27 16:22:59.278827 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.278839 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2730b6a9-a99f-4a61-b69c-74d48ee48c74-manifests\") pod \"opendatahub-operator-controller-manager-55548c94b5-9zdpp\" (UID: \"2730b6a9-a99f-4a61-b69c-74d48ee48c74\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" Jul 27 16:22:59.279056 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.278925 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp9mm\" (UniqueName: \"kubernetes.io/projected/2730b6a9-a99f-4a61-b69c-74d48ee48c74-kube-api-access-bp9mm\") pod \"opendatahub-operator-controller-manager-55548c94b5-9zdpp\" (UID: \"2730b6a9-a99f-4a61-b69c-74d48ee48c74\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" Jul 27 16:22:59.279056 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.278955 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2730b6a9-a99f-4a61-b69c-74d48ee48c74-tmp\") pod \"opendatahub-operator-controller-manager-55548c94b5-9zdpp\" (UID: \"2730b6a9-a99f-4a61-b69c-74d48ee48c74\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" Jul 27 16:22:59.279056 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.278983 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2730b6a9-a99f-4a61-b69c-74d48ee48c74-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55548c94b5-9zdpp\" (UID: \"2730b6a9-a99f-4a61-b69c-74d48ee48c74\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" Jul 27 16:22:59.379653 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.379622 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp9mm\" (UniqueName: \"kubernetes.io/projected/2730b6a9-a99f-4a61-b69c-74d48ee48c74-kube-api-access-bp9mm\") pod \"opendatahub-operator-controller-manager-55548c94b5-9zdpp\" (UID: \"2730b6a9-a99f-4a61-b69c-74d48ee48c74\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" Jul 27 16:22:59.379653 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.379655 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2730b6a9-a99f-4a61-b69c-74d48ee48c74-tmp\") pod \"opendatahub-operator-controller-manager-55548c94b5-9zdpp\" (UID: \"2730b6a9-a99f-4a61-b69c-74d48ee48c74\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" Jul 27 16:22:59.379880 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.379688 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2730b6a9-a99f-4a61-b69c-74d48ee48c74-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55548c94b5-9zdpp\" (UID: \"2730b6a9-a99f-4a61-b69c-74d48ee48c74\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" Jul 27 16:22:59.379880 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.379763 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2730b6a9-a99f-4a61-b69c-74d48ee48c74-webhook-cert\") pod \"opendatahub-operator-controller-manager-55548c94b5-9zdpp\" (UID: \"2730b6a9-a99f-4a61-b69c-74d48ee48c74\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" Jul 27 16:22:59.379880 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.379805 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2730b6a9-a99f-4a61-b69c-74d48ee48c74-manifests\") pod \"opendatahub-operator-controller-manager-55548c94b5-9zdpp\" (UID: \"2730b6a9-a99f-4a61-b69c-74d48ee48c74\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" Jul 27 16:22:59.380184 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.380161 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2730b6a9-a99f-4a61-b69c-74d48ee48c74-manifests\") pod \"opendatahub-operator-controller-manager-55548c94b5-9zdpp\" (UID: \"2730b6a9-a99f-4a61-b69c-74d48ee48c74\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" Jul 27 16:22:59.382169 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.382142 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2730b6a9-a99f-4a61-b69c-74d48ee48c74-tmp\") pod \"opendatahub-operator-controller-manager-55548c94b5-9zdpp\" (UID: \"2730b6a9-a99f-4a61-b69c-74d48ee48c74\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" Jul 27 16:22:59.382448 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.382427 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2730b6a9-a99f-4a61-b69c-74d48ee48c74-webhook-cert\") pod \"opendatahub-operator-controller-manager-55548c94b5-9zdpp\" (UID: \"2730b6a9-a99f-4a61-b69c-74d48ee48c74\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" Jul 27 16:22:59.382530 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.382474 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2730b6a9-a99f-4a61-b69c-74d48ee48c74-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55548c94b5-9zdpp\" (UID: \"2730b6a9-a99f-4a61-b69c-74d48ee48c74\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" Jul 27 16:22:59.388505 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.388487 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp9mm\" (UniqueName: \"kubernetes.io/projected/2730b6a9-a99f-4a61-b69c-74d48ee48c74-kube-api-access-bp9mm\") pod \"opendatahub-operator-controller-manager-55548c94b5-9zdpp\" (UID: \"2730b6a9-a99f-4a61-b69c-74d48ee48c74\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" Jul 27 16:22:59.527612 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.527584 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" Jul 27 16:22:59.667852 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:22:59.667749 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp"] Jul 27 16:22:59.670312 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:22:59.670285 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2730b6a9_a99f_4a61_b69c_74d48ee48c74.slice/crio-25445adb36e1c1dd40a3b1db944d1e942a56e455da23b8f0593b9955f494cbc6 WatchSource:0}: Error finding container 25445adb36e1c1dd40a3b1db944d1e942a56e455da23b8f0593b9955f494cbc6: Status 404 returned error can't find the container with id 25445adb36e1c1dd40a3b1db944d1e942a56e455da23b8f0593b9955f494cbc6 Jul 27 16:23:00.475521 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:00.475489 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" event={"ID":"2730b6a9-a99f-4a61-b69c-74d48ee48c74","Type":"ContainerStarted","Data":"25445adb36e1c1dd40a3b1db944d1e942a56e455da23b8f0593b9955f494cbc6"} Jul 27 16:23:03.495123 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:03.495085 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" event={"ID":"2730b6a9-a99f-4a61-b69c-74d48ee48c74","Type":"ContainerStarted","Data":"22b6438ccc613088e7a2129695892d8d9acee954609aef28293749d2a14734e5"} Jul 27 16:23:04.502216 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:04.502125 2550 generic.go:356] "Generic (PLEG): container finished" podID="2730b6a9-a99f-4a61-b69c-74d48ee48c74" containerID="22b6438ccc613088e7a2129695892d8d9acee954609aef28293749d2a14734e5" exitCode=0 Jul 27 16:23:04.502573 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:04.502220 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" event={"ID":"2730b6a9-a99f-4a61-b69c-74d48ee48c74","Type":"ContainerDied","Data":"22b6438ccc613088e7a2129695892d8d9acee954609aef28293749d2a14734e5"} Jul 27 16:23:05.508470 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:05.508384 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" event={"ID":"2730b6a9-a99f-4a61-b69c-74d48ee48c74","Type":"ContainerStarted","Data":"54e795fd3bcc8611fb717f32b0393be757989c86e204bcf5263b73920ce56d35"} Jul 27 16:23:05.508470 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:05.508463 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" Jul 27 16:23:05.531730 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:05.531679 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" podStartSLOduration=1.559565705 podStartE2EDuration="6.531663556s" totalImagesPullingTime="3.934511683s" totalInitContainerRuntime="1.037586168s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:22:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:23:05.530335426 +0000 UTC m=+364.202869924" watchObservedRunningTime="2026-07-27 16:23:05.531663556 +0000 UTC m=+364.204198053" Jul 27 16:23:16.515192 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:16.515163 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-9zdpp" Jul 27 16:23:21.968351 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:21.968318 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-78dbfdf94f-9trzp"] Jul 27 16:23:21.971702 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:21.971680 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-9trzp" Jul 27 16:23:21.975143 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:21.975117 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 27 16:23:21.976375 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:21.976358 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 27 16:23:21.976513 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:21.976357 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 27 16:23:21.976513 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:21.976393 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9mpv9\"" Jul 27 16:23:21.976513 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:21.976423 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 27 16:23:21.983668 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:21.983642 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78dbfdf94f-9trzp"] Jul 27 16:23:22.076209 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:22.076174 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31084f38-f413-40bf-a8ac-e5eaaa47b33a-tmp\") pod \"kube-auth-proxy-78dbfdf94f-9trzp\" (UID: \"31084f38-f413-40bf-a8ac-e5eaaa47b33a\") " pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-9trzp" Jul 27 16:23:22.076372 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:22.076244 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpn8t\" (UniqueName: \"kubernetes.io/projected/31084f38-f413-40bf-a8ac-e5eaaa47b33a-kube-api-access-hpn8t\") pod \"kube-auth-proxy-78dbfdf94f-9trzp\" (UID: \"31084f38-f413-40bf-a8ac-e5eaaa47b33a\") " pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-9trzp" Jul 27 16:23:22.076372 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:22.076302 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31084f38-f413-40bf-a8ac-e5eaaa47b33a-tls-certs\") pod \"kube-auth-proxy-78dbfdf94f-9trzp\" (UID: \"31084f38-f413-40bf-a8ac-e5eaaa47b33a\") " pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-9trzp" Jul 27 16:23:22.177660 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:22.177629 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpn8t\" (UniqueName: \"kubernetes.io/projected/31084f38-f413-40bf-a8ac-e5eaaa47b33a-kube-api-access-hpn8t\") pod \"kube-auth-proxy-78dbfdf94f-9trzp\" (UID: \"31084f38-f413-40bf-a8ac-e5eaaa47b33a\") " pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-9trzp" Jul 27 16:23:22.177825 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:22.177681 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31084f38-f413-40bf-a8ac-e5eaaa47b33a-tls-certs\") pod \"kube-auth-proxy-78dbfdf94f-9trzp\" (UID: \"31084f38-f413-40bf-a8ac-e5eaaa47b33a\") " pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-9trzp" Jul 27 16:23:22.177825 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:22.177748 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31084f38-f413-40bf-a8ac-e5eaaa47b33a-tmp\") pod \"kube-auth-proxy-78dbfdf94f-9trzp\" (UID: \"31084f38-f413-40bf-a8ac-e5eaaa47b33a\") " pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-9trzp" Jul 27 16:23:22.180225 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:22.180195 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31084f38-f413-40bf-a8ac-e5eaaa47b33a-tmp\") pod \"kube-auth-proxy-78dbfdf94f-9trzp\" (UID: \"31084f38-f413-40bf-a8ac-e5eaaa47b33a\") " pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-9trzp" Jul 27 16:23:22.180414 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:22.180397 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31084f38-f413-40bf-a8ac-e5eaaa47b33a-tls-certs\") pod \"kube-auth-proxy-78dbfdf94f-9trzp\" (UID: \"31084f38-f413-40bf-a8ac-e5eaaa47b33a\") " pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-9trzp" Jul 27 16:23:22.186613 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:22.186587 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpn8t\" (UniqueName: \"kubernetes.io/projected/31084f38-f413-40bf-a8ac-e5eaaa47b33a-kube-api-access-hpn8t\") pod \"kube-auth-proxy-78dbfdf94f-9trzp\" (UID: \"31084f38-f413-40bf-a8ac-e5eaaa47b33a\") " pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-9trzp" Jul 27 16:23:22.281171 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:22.281138 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-9trzp" Jul 27 16:23:22.421742 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:22.421720 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78dbfdf94f-9trzp"] Jul 27 16:23:22.423930 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:23:22.423889 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31084f38_f413_40bf_a8ac_e5eaaa47b33a.slice/crio-506e9001f3c0d0d2e659722fd258f44d2782847e7d291e3144ba12cc7633b691 WatchSource:0}: Error finding container 506e9001f3c0d0d2e659722fd258f44d2782847e7d291e3144ba12cc7633b691: Status 404 returned error can't find the container with id 506e9001f3c0d0d2e659722fd258f44d2782847e7d291e3144ba12cc7633b691 Jul 27 16:23:22.587153 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:22.587070 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-9trzp" event={"ID":"31084f38-f413-40bf-a8ac-e5eaaa47b33a","Type":"ContainerStarted","Data":"506e9001f3c0d0d2e659722fd258f44d2782847e7d291e3144ba12cc7633b691"} Jul 27 16:23:23.060703 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.060672 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl"] Jul 27 16:23:23.065100 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.065084 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.068367 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.068346 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 27 16:23:23.068998 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.068976 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 27 16:23:23.069213 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.069200 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 27 16:23:23.069972 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.069954 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mbr74\"" Jul 27 16:23:23.070063 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.070043 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 27 16:23:23.077765 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.077746 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl"] Jul 27 16:23:23.184919 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.184869 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.185084 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.184930 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlckv\" (UniqueName: \"kubernetes.io/projected/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-kube-api-access-zlckv\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.185084 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.184959 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.185084 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.184985 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.185084 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.185017 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.185298 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.185118 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.185298 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.185165 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.286060 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.286023 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.286263 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.286093 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.286263 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.286203 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.286263 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.286251 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlckv\" (UniqueName: \"kubernetes.io/projected/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-kube-api-access-zlckv\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.286421 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.286279 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.286421 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.286306 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.286421 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.286345 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.287183 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.286960 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.289169 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.289139 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.289264 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.289188 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.289387 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.289368 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.289467 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.289449 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.298336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.298316 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlckv\" (UniqueName: \"kubernetes.io/projected/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-kube-api-access-zlckv\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.306090 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.306045 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2q7wl\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.375662 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.375585 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:23.536810 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.536784 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl"] Jul 27 16:23:23.539502 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:23:23.539474 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97183d98_d8e1_4a93_87eb_35e5e0dacdb2.slice/crio-146479dc465ddb38d7f496bfd10de475605b4e514ff1714cc2fa48da6567a6ec WatchSource:0}: Error finding container 146479dc465ddb38d7f496bfd10de475605b4e514ff1714cc2fa48da6567a6ec: Status 404 returned error can't find the container with id 146479dc465ddb38d7f496bfd10de475605b4e514ff1714cc2fa48da6567a6ec Jul 27 16:23:23.595503 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:23.595467 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" event={"ID":"97183d98-d8e1-4a93-87eb-35e5e0dacdb2","Type":"ContainerStarted","Data":"146479dc465ddb38d7f496bfd10de475605b4e514ff1714cc2fa48da6567a6ec"} Jul 27 16:23:25.609323 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:25.609243 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-9trzp" event={"ID":"31084f38-f413-40bf-a8ac-e5eaaa47b33a","Type":"ContainerStarted","Data":"1ee0972eaa98f07b0aa0863db05a922c26729774ca97d17e2e68860b0343b9c2"} Jul 27 16:23:25.629483 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:25.629405 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-9trzp" podStartSLOduration=1.7709399380000002 podStartE2EDuration="4.629384825s" totalImagesPullingTime="2.858444887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:23:25.626975192 +0000 UTC m=+384.299509691" watchObservedRunningTime="2026-07-27 16:23:25.629384825 +0000 UTC m=+384.301919323" Jul 27 16:23:27.132233 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:27.132193 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 27 16:23:27.132596 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:27.132277 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 27 16:23:27.626423 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:27.626392 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" event={"ID":"97183d98-d8e1-4a93-87eb-35e5e0dacdb2","Type":"ContainerStarted","Data":"64e8c4ff6279ba518fe7f8caeff78960a0215e74ce0f199a561a2daa641b0394"} Jul 27 16:23:27.626623 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:27.626555 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:27.682333 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:27.682280 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" podStartSLOduration=1.09185604 podStartE2EDuration="4.682264275s" totalImagesPullingTime="3.590408235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:23:27.675385547 +0000 UTC m=+386.347920058" watchObservedRunningTime="2026-07-27 16:23:27.682264275 +0000 UTC m=+386.354798755" Jul 27 16:23:28.633582 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:28.633537 2550 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-2q7wl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 16:23:28.633954 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:28.633620 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" podUID="97183d98-d8e1-4a93-87eb-35e5e0dacdb2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 16:23:29.191687 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.191655 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d"] Jul 27 16:23:29.195187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.195171 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.198256 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.198218 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-gnpdv\"" Jul 27 16:23:29.212340 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.212317 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d"] Jul 27 16:23:29.343965 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.343930 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/022c0a96-b89f-484b-a715-6b90cc6cc408-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.343965 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.343965 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/022c0a96-b89f-484b-a715-6b90cc6cc408-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.344168 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.343988 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/022c0a96-b89f-484b-a715-6b90cc6cc408-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.344168 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.344004 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/022c0a96-b89f-484b-a715-6b90cc6cc408-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.344168 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.344081 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/022c0a96-b89f-484b-a715-6b90cc6cc408-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.344168 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.344103 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/022c0a96-b89f-484b-a715-6b90cc6cc408-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.344288 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.344173 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/022c0a96-b89f-484b-a715-6b90cc6cc408-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.344288 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.344218 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nwx9\" (UniqueName: \"kubernetes.io/projected/022c0a96-b89f-484b-a715-6b90cc6cc408-kube-api-access-4nwx9\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.344288 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.344240 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/022c0a96-b89f-484b-a715-6b90cc6cc408-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.445194 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.445108 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nwx9\" (UniqueName: \"kubernetes.io/projected/022c0a96-b89f-484b-a715-6b90cc6cc408-kube-api-access-4nwx9\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.445194 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.445147 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/022c0a96-b89f-484b-a715-6b90cc6cc408-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.445194 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.445166 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/022c0a96-b89f-484b-a715-6b90cc6cc408-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.445194 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.445191 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/022c0a96-b89f-484b-a715-6b90cc6cc408-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.445506 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.445225 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/022c0a96-b89f-484b-a715-6b90cc6cc408-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.445506 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.445250 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/022c0a96-b89f-484b-a715-6b90cc6cc408-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.445506 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.445301 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/022c0a96-b89f-484b-a715-6b90cc6cc408-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.445506 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.445322 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/022c0a96-b89f-484b-a715-6b90cc6cc408-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.445506 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.445390 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/022c0a96-b89f-484b-a715-6b90cc6cc408-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.445709 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.445608 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/022c0a96-b89f-484b-a715-6b90cc6cc408-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.445709 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.445665 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/022c0a96-b89f-484b-a715-6b90cc6cc408-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.445709 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.445702 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/022c0a96-b89f-484b-a715-6b90cc6cc408-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.445959 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.445942 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/022c0a96-b89f-484b-a715-6b90cc6cc408-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.446129 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.446082 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/022c0a96-b89f-484b-a715-6b90cc6cc408-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.447701 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.447682 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/022c0a96-b89f-484b-a715-6b90cc6cc408-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.447977 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.447959 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/022c0a96-b89f-484b-a715-6b90cc6cc408-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.459089 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.459067 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nwx9\" (UniqueName: \"kubernetes.io/projected/022c0a96-b89f-484b-a715-6b90cc6cc408-kube-api-access-4nwx9\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.461144 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.461124 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/022c0a96-b89f-484b-a715-6b90cc6cc408-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czw46d\" (UID: \"022c0a96-b89f-484b-a715-6b90cc6cc408\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.505234 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.505206 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:29.648186 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:29.648162 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d"] Jul 27 16:23:29.650471 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:23:29.650441 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022c0a96_b89f_484b_a715_6b90cc6cc408.slice/crio-8ceec853a654ef8a2ea776be9afbe12363048c8c6cbeb168df213b8a2d330f81 WatchSource:0}: Error finding container 8ceec853a654ef8a2ea776be9afbe12363048c8c6cbeb168df213b8a2d330f81: Status 404 returned error can't find the container with id 8ceec853a654ef8a2ea776be9afbe12363048c8c6cbeb168df213b8a2d330f81 Jul 27 16:23:30.643975 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:30.643938 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" event={"ID":"022c0a96-b89f-484b-a715-6b90cc6cc408","Type":"ContainerStarted","Data":"8ceec853a654ef8a2ea776be9afbe12363048c8c6cbeb168df213b8a2d330f81"} Jul 27 16:23:31.002865 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.002785 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-74lck"] Jul 27 16:23:31.009052 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.007769 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-74lck" Jul 27 16:23:31.012697 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.012669 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 27 16:23:31.013017 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.012992 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-4x2dz\"" Jul 27 16:23:31.018044 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.018021 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cpc94"] Jul 27 16:23:31.025311 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.025288 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-74lck"] Jul 27 16:23:31.025443 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.025425 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cpc94" Jul 27 16:23:31.029849 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.029816 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-lwxt2\"" Jul 27 16:23:31.030950 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.030896 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 27 16:23:31.039213 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.039179 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cpc94"] Jul 27 16:23:31.059159 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.059121 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88-cert\") pod \"kserve-controller-manager-d6f4b65b8-74lck\" (UID: \"c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-74lck" Jul 27 16:23:31.059294 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.059173 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfvjm\" (UniqueName: \"kubernetes.io/projected/c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88-kube-api-access-pfvjm\") pod \"kserve-controller-manager-d6f4b65b8-74lck\" (UID: \"c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-74lck" Jul 27 16:23:31.059294 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.059216 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/172833a1-d9d0-41f5-86a8-cf227d9a7479-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cpc94\" (UID: \"172833a1-d9d0-41f5-86a8-cf227d9a7479\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cpc94" Jul 27 16:23:31.059419 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.059365 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd4pq\" (UniqueName: \"kubernetes.io/projected/172833a1-d9d0-41f5-86a8-cf227d9a7479-kube-api-access-jd4pq\") pod \"llmisvc-controller-manager-777c6bd7d5-cpc94\" (UID: \"172833a1-d9d0-41f5-86a8-cf227d9a7479\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cpc94" Jul 27 16:23:31.160309 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.160260 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/172833a1-d9d0-41f5-86a8-cf227d9a7479-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cpc94\" (UID: \"172833a1-d9d0-41f5-86a8-cf227d9a7479\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cpc94" Jul 27 16:23:31.160478 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.160391 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd4pq\" (UniqueName: \"kubernetes.io/projected/172833a1-d9d0-41f5-86a8-cf227d9a7479-kube-api-access-jd4pq\") pod \"llmisvc-controller-manager-777c6bd7d5-cpc94\" (UID: \"172833a1-d9d0-41f5-86a8-cf227d9a7479\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cpc94" Jul 27 16:23:31.160478 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.160448 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88-cert\") pod \"kserve-controller-manager-d6f4b65b8-74lck\" (UID: \"c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-74lck" Jul 27 16:23:31.160478 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.160473 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfvjm\" (UniqueName: \"kubernetes.io/projected/c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88-kube-api-access-pfvjm\") pod \"kserve-controller-manager-d6f4b65b8-74lck\" (UID: \"c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-74lck" Jul 27 16:23:31.160705 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:23:31.160685 2550 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 27 16:23:31.160769 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:23:31.160699 2550 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 27 16:23:31.160769 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:23:31.160751 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88-cert podName:c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88 nodeName:}" failed. No retries permitted until 2026-07-27 16:23:31.660731169 +0000 UTC m=+390.333265646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88-cert") pod "kserve-controller-manager-d6f4b65b8-74lck" (UID: "c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88") : secret "kserve-webhook-server-cert" not found Jul 27 16:23:31.160891 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:23:31.160776 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/172833a1-d9d0-41f5-86a8-cf227d9a7479-cert podName:172833a1-d9d0-41f5-86a8-cf227d9a7479 nodeName:}" failed. No retries permitted until 2026-07-27 16:23:31.660758283 +0000 UTC m=+390.333292782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/172833a1-d9d0-41f5-86a8-cf227d9a7479-cert") pod "llmisvc-controller-manager-777c6bd7d5-cpc94" (UID: "172833a1-d9d0-41f5-86a8-cf227d9a7479") : secret "llmisvc-webhook-server-cert" not found Jul 27 16:23:31.178362 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.178309 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd4pq\" (UniqueName: \"kubernetes.io/projected/172833a1-d9d0-41f5-86a8-cf227d9a7479-kube-api-access-jd4pq\") pod \"llmisvc-controller-manager-777c6bd7d5-cpc94\" (UID: \"172833a1-d9d0-41f5-86a8-cf227d9a7479\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cpc94" Jul 27 16:23:31.178724 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.178699 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfvjm\" (UniqueName: \"kubernetes.io/projected/c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88-kube-api-access-pfvjm\") pod \"kserve-controller-manager-d6f4b65b8-74lck\" (UID: \"c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-74lck" Jul 27 16:23:31.632841 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.632809 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:31.665509 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.665446 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88-cert\") pod \"kserve-controller-manager-d6f4b65b8-74lck\" (UID: \"c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-74lck" Jul 27 16:23:31.665509 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.665505 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/172833a1-d9d0-41f5-86a8-cf227d9a7479-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cpc94\" (UID: \"172833a1-d9d0-41f5-86a8-cf227d9a7479\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cpc94" Jul 27 16:23:31.668876 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.668834 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/172833a1-d9d0-41f5-86a8-cf227d9a7479-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cpc94\" (UID: \"172833a1-d9d0-41f5-86a8-cf227d9a7479\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cpc94" Jul 27 16:23:31.669252 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.669216 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88-cert\") pod \"kserve-controller-manager-d6f4b65b8-74lck\" (UID: \"c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-74lck" Jul 27 16:23:31.925290 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.925198 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-74lck" Jul 27 16:23:31.939713 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:31.939688 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cpc94" Jul 27 16:23:32.376792 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.376757 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl"] Jul 27 16:23:32.377300 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.376977 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" podUID="97183d98-d8e1-4a93-87eb-35e5e0dacdb2" containerName="discovery" containerID="cri-o://64e8c4ff6279ba518fe7f8caeff78960a0215e74ce0f199a561a2daa641b0394" gracePeriod=30 Jul 27 16:23:32.658109 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.658021 2550 generic.go:356] "Generic (PLEG): container finished" podID="97183d98-d8e1-4a93-87eb-35e5e0dacdb2" containerID="64e8c4ff6279ba518fe7f8caeff78960a0215e74ce0f199a561a2daa641b0394" exitCode=0 Jul 27 16:23:32.658109 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.658061 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" event={"ID":"97183d98-d8e1-4a93-87eb-35e5e0dacdb2","Type":"ContainerDied","Data":"64e8c4ff6279ba518fe7f8caeff78960a0215e74ce0f199a561a2daa641b0394"} Jul 27 16:23:32.689369 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.689168 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 27 16:23:32.689369 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.689252 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 27 16:23:32.689369 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.689293 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 27 16:23:32.800230 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.800208 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:32.874476 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.874447 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-local-certs\") pod \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " Jul 27 16:23:32.874668 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.874511 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-csr-ca-configmap\") pod \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " Jul 27 16:23:32.874668 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.874537 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-token\") pod \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " Jul 27 16:23:32.874668 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.874647 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlckv\" (UniqueName: \"kubernetes.io/projected/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-kube-api-access-zlckv\") pod \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " Jul 27 16:23:32.874825 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.874749 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-cacerts\") pod \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " Jul 27 16:23:32.874825 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.874789 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-kubeconfig\") pod \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " Jul 27 16:23:32.874956 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.874831 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-csr-dns-cert\") pod \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\" (UID: \"97183d98-d8e1-4a93-87eb-35e5e0dacdb2\") " Jul 27 16:23:32.875010 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.874974 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "97183d98-d8e1-4a93-87eb-35e5e0dacdb2" (UID: "97183d98-d8e1-4a93-87eb-35e5e0dacdb2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:23:32.875205 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.875172 2550 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-csr-ca-configmap\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:23:32.877738 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.877701 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-token" (OuterVolumeSpecName: "istio-token") pod "97183d98-d8e1-4a93-87eb-35e5e0dacdb2" (UID: "97183d98-d8e1-4a93-87eb-35e5e0dacdb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:23:32.877738 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.877708 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-local-certs" (OuterVolumeSpecName: "local-certs") pod "97183d98-d8e1-4a93-87eb-35e5e0dacdb2" (UID: "97183d98-d8e1-4a93-87eb-35e5e0dacdb2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:23:32.878062 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.877997 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-cacerts" (OuterVolumeSpecName: "cacerts") pod "97183d98-d8e1-4a93-87eb-35e5e0dacdb2" (UID: "97183d98-d8e1-4a93-87eb-35e5e0dacdb2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 16:23:32.878062 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.878035 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "97183d98-d8e1-4a93-87eb-35e5e0dacdb2" (UID: "97183d98-d8e1-4a93-87eb-35e5e0dacdb2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 16:23:32.879699 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.879675 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "97183d98-d8e1-4a93-87eb-35e5e0dacdb2" (UID: "97183d98-d8e1-4a93-87eb-35e5e0dacdb2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 16:23:32.879781 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.879721 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-kube-api-access-zlckv" (OuterVolumeSpecName: "kube-api-access-zlckv") pod "97183d98-d8e1-4a93-87eb-35e5e0dacdb2" (UID: "97183d98-d8e1-4a93-87eb-35e5e0dacdb2"). InnerVolumeSpecName "kube-api-access-zlckv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:23:32.976485 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.976416 2550 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-kubeconfig\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:23:32.976485 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.976440 2550 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-csr-dns-cert\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:23:32.976485 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.976449 2550 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-local-certs\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:23:32.976485 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.976458 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-istio-token\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:23:32.976485 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.976467 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlckv\" (UniqueName: \"kubernetes.io/projected/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-kube-api-access-zlckv\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:23:32.976485 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:32.976477 2550 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97183d98-d8e1-4a93-87eb-35e5e0dacdb2-cacerts\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:23:33.004114 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:23:33.004087 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e5dc62_7fb0_41d6_89c5_b4d0dc541a88.slice/crio-df821f7f83d3a0e3b4c823107dd92142c4b3974a80905b5bc0a3999820098dcf WatchSource:0}: Error finding container df821f7f83d3a0e3b4c823107dd92142c4b3974a80905b5bc0a3999820098dcf: Status 404 returned error can't find the container with id df821f7f83d3a0e3b4c823107dd92142c4b3974a80905b5bc0a3999820098dcf Jul 27 16:23:33.005023 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:33.004888 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-74lck"] Jul 27 16:23:33.021928 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:33.021881 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cpc94"] Jul 27 16:23:33.024341 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:23:33.024320 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod172833a1_d9d0_41f5_86a8_cf227d9a7479.slice/crio-17d880fbb30cab006003b5163e0e990c8417adfa447e1b6d460cc8264c868fc2 WatchSource:0}: Error finding container 17d880fbb30cab006003b5163e0e990c8417adfa447e1b6d460cc8264c868fc2: Status 404 returned error can't find the container with id 17d880fbb30cab006003b5163e0e990c8417adfa447e1b6d460cc8264c868fc2 Jul 27 16:23:33.664662 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:33.664631 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cpc94" event={"ID":"172833a1-d9d0-41f5-86a8-cf227d9a7479","Type":"ContainerStarted","Data":"17d880fbb30cab006003b5163e0e990c8417adfa447e1b6d460cc8264c868fc2"} Jul 27 16:23:33.665769 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:33.665741 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-74lck" event={"ID":"c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88","Type":"ContainerStarted","Data":"df821f7f83d3a0e3b4c823107dd92142c4b3974a80905b5bc0a3999820098dcf"} Jul 27 16:23:33.667012 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:33.666988 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" event={"ID":"97183d98-d8e1-4a93-87eb-35e5e0dacdb2","Type":"ContainerDied","Data":"146479dc465ddb38d7f496bfd10de475605b4e514ff1714cc2fa48da6567a6ec"} Jul 27 16:23:33.667127 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:33.667030 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl" Jul 27 16:23:33.667127 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:33.667046 2550 scope.go:117] "RemoveContainer" containerID="64e8c4ff6279ba518fe7f8caeff78960a0215e74ce0f199a561a2daa641b0394" Jul 27 16:23:33.668658 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:33.668629 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" event={"ID":"022c0a96-b89f-484b-a715-6b90cc6cc408","Type":"ContainerStarted","Data":"116baeeed0a4ab6c861faca5648ad435337cf766e3096b24f3ff87b5c49451ef"} Jul 27 16:23:33.713104 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:33.713053 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" podStartSLOduration=1.676673202 podStartE2EDuration="4.713033136s" totalImagesPullingTime="3.036359934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:23:33.711735061 +0000 UTC m=+392.384269559" watchObservedRunningTime="2026-07-27 16:23:33.713033136 +0000 UTC m=+392.385567634" Jul 27 16:23:33.732825 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:33.732795 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl"] Jul 27 16:23:33.738245 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:33.738208 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2q7wl"] Jul 27 16:23:33.849820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:33.849778 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97183d98-d8e1-4a93-87eb-35e5e0dacdb2" path="/var/lib/kubelet/pods/97183d98-d8e1-4a93-87eb-35e5e0dacdb2/volumes" Jul 27 16:23:34.506009 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:34.505949 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:34.511877 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:34.511853 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:34.677636 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:34.677606 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:34.679183 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:34.679162 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czw46d" Jul 27 16:23:37.696113 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:37.696074 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cpc94" event={"ID":"172833a1-d9d0-41f5-86a8-cf227d9a7479","Type":"ContainerStarted","Data":"b594b8d88657745da627c2fe3fa821c4d2518e30571e7ced134c1916526f20d1"} Jul 27 16:23:37.696620 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:37.696179 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cpc94" Jul 27 16:23:37.697765 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:37.697737 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-74lck" event={"ID":"c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88","Type":"ContainerStarted","Data":"3558933be724b18c396bc2a698397cc45f6297fcbe69804e24689d359373bbfa"} Jul 27 16:23:37.697966 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:37.697946 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-74lck" Jul 27 16:23:37.717118 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:37.717070 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cpc94" podStartSLOduration=3.984865621 podStartE2EDuration="7.717057523s" totalImagesPullingTime="3.732191902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:23:37.715607925 +0000 UTC m=+396.388142423" watchObservedRunningTime="2026-07-27 16:23:37.717057523 +0000 UTC m=+396.389592020" Jul 27 16:23:37.733357 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:23:37.733314 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-74lck" podStartSLOduration=3.978764382 podStartE2EDuration="7.733300064s" totalImagesPullingTime="3.754535682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:23:37.732038299 +0000 UTC m=+396.404572799" watchObservedRunningTime="2026-07-27 16:23:37.733300064 +0000 UTC m=+396.405834562" Jul 27 16:24:08.705058 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:08.705028 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cpc94" Jul 27 16:24:08.708038 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:08.708016 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-74lck" Jul 27 16:24:19.970310 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:19.970276 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6nwww"] Jul 27 16:24:19.970734 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:19.970675 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97183d98-d8e1-4a93-87eb-35e5e0dacdb2" containerName="discovery" Jul 27 16:24:19.970734 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:19.970691 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="97183d98-d8e1-4a93-87eb-35e5e0dacdb2" containerName="discovery" Jul 27 16:24:19.970832 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:19.970811 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="97183d98-d8e1-4a93-87eb-35e5e0dacdb2" containerName="discovery" Jul 27 16:24:19.981030 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:19.980992 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6nwww" Jul 27 16:24:19.984257 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:19.984229 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-fzxr9\"" Jul 27 16:24:19.984257 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:19.984247 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 27 16:24:19.984458 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:19.984229 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 27 16:24:19.989211 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:19.989180 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6nwww"] Jul 27 16:24:20.071699 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:20.071666 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d469k\" (UniqueName: \"kubernetes.io/projected/1605464c-9600-4d7a-9951-cbe8a220ea16-kube-api-access-d469k\") pod \"kuadrant-operator-catalog-6nwww\" (UID: \"1605464c-9600-4d7a-9951-cbe8a220ea16\") " pod="kuadrant-system/kuadrant-operator-catalog-6nwww" Jul 27 16:24:20.172250 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:20.172213 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d469k\" (UniqueName: \"kubernetes.io/projected/1605464c-9600-4d7a-9951-cbe8a220ea16-kube-api-access-d469k\") pod \"kuadrant-operator-catalog-6nwww\" (UID: \"1605464c-9600-4d7a-9951-cbe8a220ea16\") " pod="kuadrant-system/kuadrant-operator-catalog-6nwww" Jul 27 16:24:20.182635 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:20.182593 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d469k\" (UniqueName: \"kubernetes.io/projected/1605464c-9600-4d7a-9951-cbe8a220ea16-kube-api-access-d469k\") pod \"kuadrant-operator-catalog-6nwww\" (UID: \"1605464c-9600-4d7a-9951-cbe8a220ea16\") " pod="kuadrant-system/kuadrant-operator-catalog-6nwww" Jul 27 16:24:20.292603 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:20.292572 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6nwww" Jul 27 16:24:20.332541 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:20.332508 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6nwww"] Jul 27 16:24:20.428421 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:20.428397 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6nwww"] Jul 27 16:24:20.430346 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:24:20.430321 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1605464c_9600_4d7a_9951_cbe8a220ea16.slice/crio-fcbd95d30e2f9628f139a8549f3257903d2c69dd55a270a65f621756cc119fd6 WatchSource:0}: Error finding container fcbd95d30e2f9628f139a8549f3257903d2c69dd55a270a65f621756cc119fd6: Status 404 returned error can't find the container with id fcbd95d30e2f9628f139a8549f3257903d2c69dd55a270a65f621756cc119fd6 Jul 27 16:24:20.544021 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:20.543944 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sckj9"] Jul 27 16:24:20.547015 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:20.547000 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sckj9" Jul 27 16:24:20.556178 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:20.556154 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sckj9"] Jul 27 16:24:20.677374 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:20.677344 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fppgh\" (UniqueName: \"kubernetes.io/projected/f3122b10-967f-4583-bdc0-9fb1e8ce81e5-kube-api-access-fppgh\") pod \"kuadrant-operator-catalog-sckj9\" (UID: \"f3122b10-967f-4583-bdc0-9fb1e8ce81e5\") " pod="kuadrant-system/kuadrant-operator-catalog-sckj9" Jul 27 16:24:20.778747 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:20.778712 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fppgh\" (UniqueName: \"kubernetes.io/projected/f3122b10-967f-4583-bdc0-9fb1e8ce81e5-kube-api-access-fppgh\") pod \"kuadrant-operator-catalog-sckj9\" (UID: \"f3122b10-967f-4583-bdc0-9fb1e8ce81e5\") " pod="kuadrant-system/kuadrant-operator-catalog-sckj9" Jul 27 16:24:20.788194 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:20.788167 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fppgh\" (UniqueName: \"kubernetes.io/projected/f3122b10-967f-4583-bdc0-9fb1e8ce81e5-kube-api-access-fppgh\") pod \"kuadrant-operator-catalog-sckj9\" (UID: \"f3122b10-967f-4583-bdc0-9fb1e8ce81e5\") " pod="kuadrant-system/kuadrant-operator-catalog-sckj9" Jul 27 16:24:20.857848 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:20.857767 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sckj9" Jul 27 16:24:20.916305 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:20.916276 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6nwww" event={"ID":"1605464c-9600-4d7a-9951-cbe8a220ea16","Type":"ContainerStarted","Data":"fcbd95d30e2f9628f139a8549f3257903d2c69dd55a270a65f621756cc119fd6"} Jul 27 16:24:20.993484 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:20.993458 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sckj9"] Jul 27 16:24:20.995372 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:24:20.995339 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3122b10_967f_4583_bdc0_9fb1e8ce81e5.slice/crio-9baf371bfc523cfe3703e69f6116e7fd124d67deaafd817e38d571e94c6a5741 WatchSource:0}: Error finding container 9baf371bfc523cfe3703e69f6116e7fd124d67deaafd817e38d571e94c6a5741: Status 404 returned error can't find the container with id 9baf371bfc523cfe3703e69f6116e7fd124d67deaafd817e38d571e94c6a5741 Jul 27 16:24:21.923224 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:21.923183 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sckj9" event={"ID":"f3122b10-967f-4583-bdc0-9fb1e8ce81e5","Type":"ContainerStarted","Data":"9baf371bfc523cfe3703e69f6116e7fd124d67deaafd817e38d571e94c6a5741"} Jul 27 16:24:23.935101 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:23.935004 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6nwww" event={"ID":"1605464c-9600-4d7a-9951-cbe8a220ea16","Type":"ContainerStarted","Data":"bf108c3aa9e9d7b902b6c6fa17dc7e927ca927e114fc642fa471b2f63253653a"} Jul 27 16:24:23.935505 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:23.935093 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-6nwww" podUID="1605464c-9600-4d7a-9951-cbe8a220ea16" containerName="registry-server" containerID="cri-o://bf108c3aa9e9d7b902b6c6fa17dc7e927ca927e114fc642fa471b2f63253653a" gracePeriod=2 Jul 27 16:24:23.936709 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:23.936685 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sckj9" event={"ID":"f3122b10-967f-4583-bdc0-9fb1e8ce81e5","Type":"ContainerStarted","Data":"421500c5f115f6af0203bb22a279e824bcfa2c568bcc4302a725da132c9eb8cc"} Jul 27 16:24:23.957082 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:23.957038 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-6nwww" podStartSLOduration=1.8388240790000001 podStartE2EDuration="4.957022496s" totalImagesPullingTime="3.118198417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:24:23.954332384 +0000 UTC m=+442.626866884" watchObservedRunningTime="2026-07-27 16:24:23.957022496 +0000 UTC m=+442.629556995" Jul 27 16:24:23.971993 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:23.971951 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-sckj9" podStartSLOduration=1.4217977290000001 podStartE2EDuration="3.971936607s" totalImagesPullingTime="2.550138878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:24:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:24:23.969795078 +0000 UTC m=+442.642329577" watchObservedRunningTime="2026-07-27 16:24:23.971936607 +0000 UTC m=+442.644471105" Jul 27 16:24:24.182464 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:24.182436 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6nwww" Jul 27 16:24:24.315868 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:24.315834 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d469k\" (UniqueName: \"kubernetes.io/projected/1605464c-9600-4d7a-9951-cbe8a220ea16-kube-api-access-d469k\") pod \"1605464c-9600-4d7a-9951-cbe8a220ea16\" (UID: \"1605464c-9600-4d7a-9951-cbe8a220ea16\") " Jul 27 16:24:24.318143 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:24.318112 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1605464c-9600-4d7a-9951-cbe8a220ea16-kube-api-access-d469k" (OuterVolumeSpecName: "kube-api-access-d469k") pod "1605464c-9600-4d7a-9951-cbe8a220ea16" (UID: "1605464c-9600-4d7a-9951-cbe8a220ea16"). InnerVolumeSpecName "kube-api-access-d469k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:24:24.416748 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:24.416715 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d469k\" (UniqueName: \"kubernetes.io/projected/1605464c-9600-4d7a-9951-cbe8a220ea16-kube-api-access-d469k\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:24:24.943857 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:24.943754 2550 generic.go:356] "Generic (PLEG): container finished" podID="1605464c-9600-4d7a-9951-cbe8a220ea16" containerID="bf108c3aa9e9d7b902b6c6fa17dc7e927ca927e114fc642fa471b2f63253653a" exitCode=0 Jul 27 16:24:24.943857 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:24.943811 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6nwww" Jul 27 16:24:24.943857 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:24.943826 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6nwww" event={"ID":"1605464c-9600-4d7a-9951-cbe8a220ea16","Type":"ContainerDied","Data":"bf108c3aa9e9d7b902b6c6fa17dc7e927ca927e114fc642fa471b2f63253653a"} Jul 27 16:24:24.944426 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:24.943898 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6nwww" event={"ID":"1605464c-9600-4d7a-9951-cbe8a220ea16","Type":"ContainerDied","Data":"fcbd95d30e2f9628f139a8549f3257903d2c69dd55a270a65f621756cc119fd6"} Jul 27 16:24:24.944426 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:24.943939 2550 scope.go:117] "RemoveContainer" containerID="bf108c3aa9e9d7b902b6c6fa17dc7e927ca927e114fc642fa471b2f63253653a" Jul 27 16:24:24.955404 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:24.955386 2550 scope.go:117] "RemoveContainer" containerID="bf108c3aa9e9d7b902b6c6fa17dc7e927ca927e114fc642fa471b2f63253653a" Jul 27 16:24:24.955816 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:24:24.955795 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf108c3aa9e9d7b902b6c6fa17dc7e927ca927e114fc642fa471b2f63253653a\": container with ID starting with bf108c3aa9e9d7b902b6c6fa17dc7e927ca927e114fc642fa471b2f63253653a not found: ID does not exist" containerID="bf108c3aa9e9d7b902b6c6fa17dc7e927ca927e114fc642fa471b2f63253653a" Jul 27 16:24:24.955897 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:24.955823 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf108c3aa9e9d7b902b6c6fa17dc7e927ca927e114fc642fa471b2f63253653a"} err="failed to get container status \"bf108c3aa9e9d7b902b6c6fa17dc7e927ca927e114fc642fa471b2f63253653a\": rpc error: code = NotFound desc = could not find container \"bf108c3aa9e9d7b902b6c6fa17dc7e927ca927e114fc642fa471b2f63253653a\": container with ID starting with bf108c3aa9e9d7b902b6c6fa17dc7e927ca927e114fc642fa471b2f63253653a not found: ID does not exist" Jul 27 16:24:24.967405 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:24.967374 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6nwww"] Jul 27 16:24:24.974665 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:24.974642 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6nwww"] Jul 27 16:24:25.853259 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:25.853227 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1605464c-9600-4d7a-9951-cbe8a220ea16" path="/var/lib/kubelet/pods/1605464c-9600-4d7a-9951-cbe8a220ea16/volumes" Jul 27 16:24:30.857894 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:30.857861 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-sckj9" Jul 27 16:24:30.858346 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:30.857945 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-sckj9" Jul 27 16:24:30.880178 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:30.880153 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-sckj9" Jul 27 16:24:31.001698 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:31.001677 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-sckj9" Jul 27 16:24:35.369339 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.369298 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm"] Jul 27 16:24:35.369805 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.369673 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1605464c-9600-4d7a-9951-cbe8a220ea16" containerName="registry-server" Jul 27 16:24:35.369805 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.369685 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1605464c-9600-4d7a-9951-cbe8a220ea16" containerName="registry-server" Jul 27 16:24:35.369805 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.369755 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1605464c-9600-4d7a-9951-cbe8a220ea16" containerName="registry-server" Jul 27 16:24:35.373251 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.373232 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm" Jul 27 16:24:35.378564 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.378547 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2xkwb\"" Jul 27 16:24:35.392420 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.392378 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm"] Jul 27 16:24:35.517917 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.517866 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25a355db-00df-4c8f-82f1-13a486427ea3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm\" (UID: \"25a355db-00df-4c8f-82f1-13a486427ea3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm" Jul 27 16:24:35.518106 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.517934 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p85vh\" (UniqueName: \"kubernetes.io/projected/25a355db-00df-4c8f-82f1-13a486427ea3-kube-api-access-p85vh\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm\" (UID: \"25a355db-00df-4c8f-82f1-13a486427ea3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm" Jul 27 16:24:35.518106 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.517967 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25a355db-00df-4c8f-82f1-13a486427ea3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm\" (UID: \"25a355db-00df-4c8f-82f1-13a486427ea3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm" Jul 27 16:24:35.618821 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.618779 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25a355db-00df-4c8f-82f1-13a486427ea3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm\" (UID: \"25a355db-00df-4c8f-82f1-13a486427ea3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm" Jul 27 16:24:35.618821 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.618821 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p85vh\" (UniqueName: \"kubernetes.io/projected/25a355db-00df-4c8f-82f1-13a486427ea3-kube-api-access-p85vh\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm\" (UID: \"25a355db-00df-4c8f-82f1-13a486427ea3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm" Jul 27 16:24:35.619117 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.618851 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25a355db-00df-4c8f-82f1-13a486427ea3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm\" (UID: \"25a355db-00df-4c8f-82f1-13a486427ea3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm" Jul 27 16:24:35.619276 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.619254 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25a355db-00df-4c8f-82f1-13a486427ea3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm\" (UID: \"25a355db-00df-4c8f-82f1-13a486427ea3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm" Jul 27 16:24:35.619333 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.619272 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25a355db-00df-4c8f-82f1-13a486427ea3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm\" (UID: \"25a355db-00df-4c8f-82f1-13a486427ea3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm" Jul 27 16:24:35.629261 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.629188 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p85vh\" (UniqueName: \"kubernetes.io/projected/25a355db-00df-4c8f-82f1-13a486427ea3-kube-api-access-p85vh\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm\" (UID: \"25a355db-00df-4c8f-82f1-13a486427ea3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm" Jul 27 16:24:35.682395 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.682355 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm" Jul 27 16:24:35.837496 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.837466 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm"] Jul 27 16:24:35.841007 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:24:35.840981 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a355db_00df_4c8f_82f1_13a486427ea3.slice/crio-cf81f47e7ccf23d918c5fe62d7be18e442cca5a240cc98f74faf8e11fc0b78da WatchSource:0}: Error finding container cf81f47e7ccf23d918c5fe62d7be18e442cca5a240cc98f74faf8e11fc0b78da: Status 404 returned error can't find the container with id cf81f47e7ccf23d918c5fe62d7be18e442cca5a240cc98f74faf8e11fc0b78da Jul 27 16:24:35.988031 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.987994 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl"] Jul 27 16:24:35.991011 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:35.990987 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl" Jul 27 16:24:36.008301 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.007349 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl"] Jul 27 16:24:36.015321 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.015264 2550 generic.go:356] "Generic (PLEG): container finished" podID="25a355db-00df-4c8f-82f1-13a486427ea3" containerID="35e11bbf46b143f3d387fcee8fa0831beba9f1f4c99aa4925cea7ad149071c06" exitCode=0 Jul 27 16:24:36.015465 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.015358 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm" event={"ID":"25a355db-00df-4c8f-82f1-13a486427ea3","Type":"ContainerDied","Data":"35e11bbf46b143f3d387fcee8fa0831beba9f1f4c99aa4925cea7ad149071c06"} Jul 27 16:24:36.015465 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.015451 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm" event={"ID":"25a355db-00df-4c8f-82f1-13a486427ea3","Type":"ContainerStarted","Data":"cf81f47e7ccf23d918c5fe62d7be18e442cca5a240cc98f74faf8e11fc0b78da"} Jul 27 16:24:36.122701 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.122655 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6fd183d-09c5-43b3-894e-99a5c0c4c6d4-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl\" (UID: \"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl" Jul 27 16:24:36.122872 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.122778 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6fd183d-09c5-43b3-894e-99a5c0c4c6d4-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl\" (UID: \"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl" Jul 27 16:24:36.122872 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.122831 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbnn6\" (UniqueName: \"kubernetes.io/projected/e6fd183d-09c5-43b3-894e-99a5c0c4c6d4-kube-api-access-qbnn6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl\" (UID: \"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl" Jul 27 16:24:36.223768 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.223665 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6fd183d-09c5-43b3-894e-99a5c0c4c6d4-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl\" (UID: \"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl" Jul 27 16:24:36.223768 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.223722 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6fd183d-09c5-43b3-894e-99a5c0c4c6d4-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl\" (UID: \"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl" Jul 27 16:24:36.223768 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.223775 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbnn6\" (UniqueName: \"kubernetes.io/projected/e6fd183d-09c5-43b3-894e-99a5c0c4c6d4-kube-api-access-qbnn6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl\" (UID: \"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl" Jul 27 16:24:36.224129 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.224104 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6fd183d-09c5-43b3-894e-99a5c0c4c6d4-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl\" (UID: \"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl" Jul 27 16:24:36.224176 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.224165 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6fd183d-09c5-43b3-894e-99a5c0c4c6d4-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl\" (UID: \"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl" Jul 27 16:24:36.233545 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.233518 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbnn6\" (UniqueName: \"kubernetes.io/projected/e6fd183d-09c5-43b3-894e-99a5c0c4c6d4-kube-api-access-qbnn6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl\" (UID: \"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl" Jul 27 16:24:36.309543 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.309496 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl" Jul 27 16:24:36.372642 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.372605 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p"] Jul 27 16:24:36.376408 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.376379 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" Jul 27 16:24:36.387079 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.387052 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p"] Jul 27 16:24:36.463257 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.463227 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl"] Jul 27 16:24:36.464924 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:24:36.464866 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6fd183d_09c5_43b3_894e_99a5c0c4c6d4.slice/crio-0cd327f719aa49593947ee15412bca81bb5c8ee3e4506ae3bcc3116a3a3c10f9 WatchSource:0}: Error finding container 0cd327f719aa49593947ee15412bca81bb5c8ee3e4506ae3bcc3116a3a3c10f9: Status 404 returned error can't find the container with id 0cd327f719aa49593947ee15412bca81bb5c8ee3e4506ae3bcc3116a3a3c10f9 Jul 27 16:24:36.527690 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.527656 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8zbs\" (UniqueName: \"kubernetes.io/projected/f55fb6fb-63db-4382-bb60-1ac2a9bc7a14-kube-api-access-g8zbs\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p\" (UID: \"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" Jul 27 16:24:36.527867 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.527722 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f55fb6fb-63db-4382-bb60-1ac2a9bc7a14-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p\" (UID: \"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" Jul 27 16:24:36.527961 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.527942 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f55fb6fb-63db-4382-bb60-1ac2a9bc7a14-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p\" (UID: \"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" Jul 27 16:24:36.629026 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.628986 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f55fb6fb-63db-4382-bb60-1ac2a9bc7a14-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p\" (UID: \"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" Jul 27 16:24:36.629243 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.629053 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8zbs\" (UniqueName: \"kubernetes.io/projected/f55fb6fb-63db-4382-bb60-1ac2a9bc7a14-kube-api-access-g8zbs\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p\" (UID: \"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" Jul 27 16:24:36.629243 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.629093 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f55fb6fb-63db-4382-bb60-1ac2a9bc7a14-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p\" (UID: \"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" Jul 27 16:24:36.629515 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.629427 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f55fb6fb-63db-4382-bb60-1ac2a9bc7a14-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p\" (UID: \"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" Jul 27 16:24:36.629515 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.629443 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f55fb6fb-63db-4382-bb60-1ac2a9bc7a14-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p\" (UID: \"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" Jul 27 16:24:36.640136 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.640104 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8zbs\" (UniqueName: \"kubernetes.io/projected/f55fb6fb-63db-4382-bb60-1ac2a9bc7a14-kube-api-access-g8zbs\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p\" (UID: \"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" Jul 27 16:24:36.687008 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.686974 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" Jul 27 16:24:36.775131 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.775100 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz"] Jul 27 16:24:36.778322 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.778301 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz" Jul 27 16:24:36.789651 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.789620 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz"] Jul 27 16:24:36.910525 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.910501 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p"] Jul 27 16:24:36.932548 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.932509 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe048780-6d3f-4ba5-b296-ef02c3fee70c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz\" (UID: \"fe048780-6d3f-4ba5-b296-ef02c3fee70c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz" Jul 27 16:24:36.932734 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.932575 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gklp9\" (UniqueName: \"kubernetes.io/projected/fe048780-6d3f-4ba5-b296-ef02c3fee70c-kube-api-access-gklp9\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz\" (UID: \"fe048780-6d3f-4ba5-b296-ef02c3fee70c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz" Jul 27 16:24:36.932734 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:36.932619 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe048780-6d3f-4ba5-b296-ef02c3fee70c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz\" (UID: \"fe048780-6d3f-4ba5-b296-ef02c3fee70c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz" Jul 27 16:24:36.937915 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:24:36.937865 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55fb6fb_63db_4382_bb60_1ac2a9bc7a14.slice/crio-c0feac8c7d1f8be757225d8caf6422fd3c8c4a6f76b1a1331de654cad9cde917 WatchSource:0}: Error finding container c0feac8c7d1f8be757225d8caf6422fd3c8c4a6f76b1a1331de654cad9cde917: Status 404 returned error can't find the container with id c0feac8c7d1f8be757225d8caf6422fd3c8c4a6f76b1a1331de654cad9cde917 Jul 27 16:24:37.023771 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:37.023739 2550 generic.go:356] "Generic (PLEG): container finished" podID="e6fd183d-09c5-43b3-894e-99a5c0c4c6d4" containerID="00847a62fee20c32015cc3344043d4f5693073182db05353bf88b21a4f34c06a" exitCode=0 Jul 27 16:24:37.023949 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:37.023801 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl" event={"ID":"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4","Type":"ContainerDied","Data":"00847a62fee20c32015cc3344043d4f5693073182db05353bf88b21a4f34c06a"} Jul 27 16:24:37.023949 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:37.023856 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl" event={"ID":"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4","Type":"ContainerStarted","Data":"0cd327f719aa49593947ee15412bca81bb5c8ee3e4506ae3bcc3116a3a3c10f9"} Jul 27 16:24:37.026140 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:37.026111 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" event={"ID":"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14","Type":"ContainerStarted","Data":"f14e55dd59651c14915087978adf7a033c58648a4869cb9f6ba2ada4ffbb0abd"} Jul 27 16:24:37.026280 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:37.026163 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" event={"ID":"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14","Type":"ContainerStarted","Data":"c0feac8c7d1f8be757225d8caf6422fd3c8c4a6f76b1a1331de654cad9cde917"} Jul 27 16:24:37.028470 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:37.028408 2550 generic.go:356] "Generic (PLEG): container finished" podID="25a355db-00df-4c8f-82f1-13a486427ea3" containerID="710626d056ea9473e4f70c4c86f9307b18076899987f8aa5e58d5c3ae4a6430d" exitCode=0 Jul 27 16:24:37.028470 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:37.028452 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm" event={"ID":"25a355db-00df-4c8f-82f1-13a486427ea3","Type":"ContainerDied","Data":"710626d056ea9473e4f70c4c86f9307b18076899987f8aa5e58d5c3ae4a6430d"} Jul 27 16:24:37.033161 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:37.033136 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gklp9\" (UniqueName: \"kubernetes.io/projected/fe048780-6d3f-4ba5-b296-ef02c3fee70c-kube-api-access-gklp9\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz\" (UID: \"fe048780-6d3f-4ba5-b296-ef02c3fee70c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz" Jul 27 16:24:37.033269 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:37.033231 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe048780-6d3f-4ba5-b296-ef02c3fee70c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz\" (UID: \"fe048780-6d3f-4ba5-b296-ef02c3fee70c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz" Jul 27 16:24:37.033409 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:37.033363 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe048780-6d3f-4ba5-b296-ef02c3fee70c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz\" (UID: \"fe048780-6d3f-4ba5-b296-ef02c3fee70c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz" Jul 27 16:24:37.033820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:37.033791 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe048780-6d3f-4ba5-b296-ef02c3fee70c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz\" (UID: \"fe048780-6d3f-4ba5-b296-ef02c3fee70c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz" Jul 27 16:24:37.033820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:37.033812 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe048780-6d3f-4ba5-b296-ef02c3fee70c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz\" (UID: \"fe048780-6d3f-4ba5-b296-ef02c3fee70c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz" Jul 27 16:24:37.046438 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:37.046411 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gklp9\" (UniqueName: \"kubernetes.io/projected/fe048780-6d3f-4ba5-b296-ef02c3fee70c-kube-api-access-gklp9\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz\" (UID: \"fe048780-6d3f-4ba5-b296-ef02c3fee70c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz" Jul 27 16:24:37.091520 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:37.091496 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz" Jul 27 16:24:37.254697 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:37.254668 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz"] Jul 27 16:24:37.256127 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:24:37.256103 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe048780_6d3f_4ba5_b296_ef02c3fee70c.slice/crio-f1aed7fd09045088dce4068793966463b2afd799b8f33415a5b4aedeaed28a8b WatchSource:0}: Error finding container f1aed7fd09045088dce4068793966463b2afd799b8f33415a5b4aedeaed28a8b: Status 404 returned error can't find the container with id f1aed7fd09045088dce4068793966463b2afd799b8f33415a5b4aedeaed28a8b Jul 27 16:24:38.035206 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:38.035170 2550 generic.go:356] "Generic (PLEG): container finished" podID="e6fd183d-09c5-43b3-894e-99a5c0c4c6d4" containerID="21be02d2aaaa65174930f22614ff32971b8aa94c7feb34bd83f877f9cb8bcc64" exitCode=0 Jul 27 16:24:38.035665 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:38.035238 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl" event={"ID":"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4","Type":"ContainerDied","Data":"21be02d2aaaa65174930f22614ff32971b8aa94c7feb34bd83f877f9cb8bcc64"} Jul 27 16:24:38.036935 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:38.036893 2550 generic.go:356] "Generic (PLEG): container finished" podID="fe048780-6d3f-4ba5-b296-ef02c3fee70c" containerID="aaae5e1a80f81996a2de0f936594799e196febb6b170650b8bfb919c96a94807" exitCode=0 Jul 27 16:24:38.037055 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:38.036936 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz" event={"ID":"fe048780-6d3f-4ba5-b296-ef02c3fee70c","Type":"ContainerDied","Data":"aaae5e1a80f81996a2de0f936594799e196febb6b170650b8bfb919c96a94807"} Jul 27 16:24:38.037055 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:38.036996 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz" event={"ID":"fe048780-6d3f-4ba5-b296-ef02c3fee70c","Type":"ContainerStarted","Data":"f1aed7fd09045088dce4068793966463b2afd799b8f33415a5b4aedeaed28a8b"} Jul 27 16:24:38.038642 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:38.038622 2550 generic.go:356] "Generic (PLEG): container finished" podID="f55fb6fb-63db-4382-bb60-1ac2a9bc7a14" containerID="f14e55dd59651c14915087978adf7a033c58648a4869cb9f6ba2ada4ffbb0abd" exitCode=0 Jul 27 16:24:38.038738 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:38.038704 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" event={"ID":"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14","Type":"ContainerDied","Data":"f14e55dd59651c14915087978adf7a033c58648a4869cb9f6ba2ada4ffbb0abd"} Jul 27 16:24:38.041336 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:38.041313 2550 generic.go:356] "Generic (PLEG): container finished" podID="25a355db-00df-4c8f-82f1-13a486427ea3" containerID="7fe2be344cd4938018712772ffd913bbb7a22a12fb330f87e76d0d718c69457d" exitCode=0 Jul 27 16:24:38.041420 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:38.041362 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm" event={"ID":"25a355db-00df-4c8f-82f1-13a486427ea3","Type":"ContainerDied","Data":"7fe2be344cd4938018712772ffd913bbb7a22a12fb330f87e76d0d718c69457d"} Jul 27 16:24:39.050366 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:39.050332 2550 generic.go:356] "Generic (PLEG): container finished" podID="e6fd183d-09c5-43b3-894e-99a5c0c4c6d4" containerID="2f8a9600ff30bc6cf5c636c976e6fce848552c1e2698513e44418018b06d8357" exitCode=0 Jul 27 16:24:39.050814 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:39.050422 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl" event={"ID":"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4","Type":"ContainerDied","Data":"2f8a9600ff30bc6cf5c636c976e6fce848552c1e2698513e44418018b06d8357"} Jul 27 16:24:39.186342 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:39.186310 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm" Jul 27 16:24:39.255681 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:39.255647 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25a355db-00df-4c8f-82f1-13a486427ea3-bundle\") pod \"25a355db-00df-4c8f-82f1-13a486427ea3\" (UID: \"25a355db-00df-4c8f-82f1-13a486427ea3\") " Jul 27 16:24:39.255681 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:39.255689 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p85vh\" (UniqueName: \"kubernetes.io/projected/25a355db-00df-4c8f-82f1-13a486427ea3-kube-api-access-p85vh\") pod \"25a355db-00df-4c8f-82f1-13a486427ea3\" (UID: \"25a355db-00df-4c8f-82f1-13a486427ea3\") " Jul 27 16:24:39.255927 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:39.255709 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25a355db-00df-4c8f-82f1-13a486427ea3-util\") pod \"25a355db-00df-4c8f-82f1-13a486427ea3\" (UID: \"25a355db-00df-4c8f-82f1-13a486427ea3\") " Jul 27 16:24:39.256317 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:39.256288 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a355db-00df-4c8f-82f1-13a486427ea3-bundle" (OuterVolumeSpecName: "bundle") pod "25a355db-00df-4c8f-82f1-13a486427ea3" (UID: "25a355db-00df-4c8f-82f1-13a486427ea3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:24:39.258101 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:39.258071 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a355db-00df-4c8f-82f1-13a486427ea3-kube-api-access-p85vh" (OuterVolumeSpecName: "kube-api-access-p85vh") pod "25a355db-00df-4c8f-82f1-13a486427ea3" (UID: "25a355db-00df-4c8f-82f1-13a486427ea3"). InnerVolumeSpecName "kube-api-access-p85vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:24:39.260691 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:39.260662 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a355db-00df-4c8f-82f1-13a486427ea3-util" (OuterVolumeSpecName: "util") pod "25a355db-00df-4c8f-82f1-13a486427ea3" (UID: "25a355db-00df-4c8f-82f1-13a486427ea3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:24:39.356665 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:39.356576 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25a355db-00df-4c8f-82f1-13a486427ea3-bundle\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:24:39.356665 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:39.356606 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p85vh\" (UniqueName: \"kubernetes.io/projected/25a355db-00df-4c8f-82f1-13a486427ea3-kube-api-access-p85vh\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:24:39.356665 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:39.356617 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25a355db-00df-4c8f-82f1-13a486427ea3-util\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:24:40.058401 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:40.058366 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm" Jul 27 16:24:40.058857 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:40.058400 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm" event={"ID":"25a355db-00df-4c8f-82f1-13a486427ea3","Type":"ContainerDied","Data":"cf81f47e7ccf23d918c5fe62d7be18e442cca5a240cc98f74faf8e11fc0b78da"} Jul 27 16:24:40.058857 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:40.058453 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf81f47e7ccf23d918c5fe62d7be18e442cca5a240cc98f74faf8e11fc0b78da" Jul 27 16:24:40.196550 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:40.196525 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl" Jul 27 16:24:40.262779 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:40.262746 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbnn6\" (UniqueName: \"kubernetes.io/projected/e6fd183d-09c5-43b3-894e-99a5c0c4c6d4-kube-api-access-qbnn6\") pod \"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4\" (UID: \"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4\") " Jul 27 16:24:40.262993 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:40.262794 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6fd183d-09c5-43b3-894e-99a5c0c4c6d4-bundle\") pod \"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4\" (UID: \"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4\") " Jul 27 16:24:40.262993 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:40.262828 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6fd183d-09c5-43b3-894e-99a5c0c4c6d4-util\") pod \"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4\" (UID: \"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4\") " Jul 27 16:24:40.263379 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:40.263344 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fd183d-09c5-43b3-894e-99a5c0c4c6d4-bundle" (OuterVolumeSpecName: "bundle") pod "e6fd183d-09c5-43b3-894e-99a5c0c4c6d4" (UID: "e6fd183d-09c5-43b3-894e-99a5c0c4c6d4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:24:40.265139 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:40.265114 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fd183d-09c5-43b3-894e-99a5c0c4c6d4-kube-api-access-qbnn6" (OuterVolumeSpecName: "kube-api-access-qbnn6") pod "e6fd183d-09c5-43b3-894e-99a5c0c4c6d4" (UID: "e6fd183d-09c5-43b3-894e-99a5c0c4c6d4"). InnerVolumeSpecName "kube-api-access-qbnn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:24:40.269881 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:40.269846 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fd183d-09c5-43b3-894e-99a5c0c4c6d4-util" (OuterVolumeSpecName: "util") pod "e6fd183d-09c5-43b3-894e-99a5c0c4c6d4" (UID: "e6fd183d-09c5-43b3-894e-99a5c0c4c6d4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:24:40.364231 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:40.364187 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbnn6\" (UniqueName: \"kubernetes.io/projected/e6fd183d-09c5-43b3-894e-99a5c0c4c6d4-kube-api-access-qbnn6\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:24:40.364231 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:40.364221 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6fd183d-09c5-43b3-894e-99a5c0c4c6d4-bundle\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:24:40.364231 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:40.364230 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6fd183d-09c5-43b3-894e-99a5c0c4c6d4-util\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:24:41.065701 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:41.065675 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl" Jul 27 16:24:41.065701 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:41.065692 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl" event={"ID":"e6fd183d-09c5-43b3-894e-99a5c0c4c6d4","Type":"ContainerDied","Data":"0cd327f719aa49593947ee15412bca81bb5c8ee3e4506ae3bcc3116a3a3c10f9"} Jul 27 16:24:41.066177 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:41.065746 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cd327f719aa49593947ee15412bca81bb5c8ee3e4506ae3bcc3116a3a3c10f9" Jul 27 16:24:41.067805 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:41.067782 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" event={"ID":"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14","Type":"ContainerStarted","Data":"e9f977c0cc30f568483692260c5e213239f1d04754f998262002fcf482fb281e"} Jul 27 16:24:42.075094 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:42.075060 2550 generic.go:356] "Generic (PLEG): container finished" podID="f55fb6fb-63db-4382-bb60-1ac2a9bc7a14" containerID="e9f977c0cc30f568483692260c5e213239f1d04754f998262002fcf482fb281e" exitCode=0 Jul 27 16:24:42.075551 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:42.075141 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" event={"ID":"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14","Type":"ContainerDied","Data":"e9f977c0cc30f568483692260c5e213239f1d04754f998262002fcf482fb281e"} Jul 27 16:24:42.077096 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:42.077072 2550 generic.go:356] "Generic (PLEG): container finished" podID="fe048780-6d3f-4ba5-b296-ef02c3fee70c" containerID="177a5d80e9798e9ddf4119293eec316a5573ab9b872a8db32332d0fe3cac0c60" exitCode=0 Jul 27 16:24:42.077194 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:42.077106 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz" event={"ID":"fe048780-6d3f-4ba5-b296-ef02c3fee70c","Type":"ContainerDied","Data":"177a5d80e9798e9ddf4119293eec316a5573ab9b872a8db32332d0fe3cac0c60"} Jul 27 16:24:43.084138 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:43.084106 2550 generic.go:356] "Generic (PLEG): container finished" podID="fe048780-6d3f-4ba5-b296-ef02c3fee70c" containerID="989ad27e1fadb5d1223736879d6fcf049e81db73e45b94b23b8fd9b1a2c6a1a3" exitCode=0 Jul 27 16:24:43.084591 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:43.084193 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz" event={"ID":"fe048780-6d3f-4ba5-b296-ef02c3fee70c","Type":"ContainerDied","Data":"989ad27e1fadb5d1223736879d6fcf049e81db73e45b94b23b8fd9b1a2c6a1a3"} Jul 27 16:24:43.086210 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:43.086180 2550 generic.go:356] "Generic (PLEG): container finished" podID="f55fb6fb-63db-4382-bb60-1ac2a9bc7a14" containerID="370eea46e9ce93291e22f60d194ea0a708bc5b410aa6bbdee24329097a408622" exitCode=0 Jul 27 16:24:43.086303 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:43.086240 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" event={"ID":"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14","Type":"ContainerDied","Data":"370eea46e9ce93291e22f60d194ea0a708bc5b410aa6bbdee24329097a408622"} Jul 27 16:24:44.235876 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.235850 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" Jul 27 16:24:44.266923 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.266873 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz" Jul 27 16:24:44.298596 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.298568 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f55fb6fb-63db-4382-bb60-1ac2a9bc7a14-util\") pod \"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14\" (UID: \"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14\") " Jul 27 16:24:44.298777 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.298606 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8zbs\" (UniqueName: \"kubernetes.io/projected/f55fb6fb-63db-4382-bb60-1ac2a9bc7a14-kube-api-access-g8zbs\") pod \"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14\" (UID: \"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14\") " Jul 27 16:24:44.298777 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.298709 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f55fb6fb-63db-4382-bb60-1ac2a9bc7a14-bundle\") pod \"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14\" (UID: \"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14\") " Jul 27 16:24:44.298777 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.298742 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe048780-6d3f-4ba5-b296-ef02c3fee70c-util\") pod \"fe048780-6d3f-4ba5-b296-ef02c3fee70c\" (UID: \"fe048780-6d3f-4ba5-b296-ef02c3fee70c\") " Jul 27 16:24:44.298950 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.298826 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gklp9\" (UniqueName: \"kubernetes.io/projected/fe048780-6d3f-4ba5-b296-ef02c3fee70c-kube-api-access-gklp9\") pod \"fe048780-6d3f-4ba5-b296-ef02c3fee70c\" (UID: \"fe048780-6d3f-4ba5-b296-ef02c3fee70c\") " Jul 27 16:24:44.298950 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.298888 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe048780-6d3f-4ba5-b296-ef02c3fee70c-bundle\") pod \"fe048780-6d3f-4ba5-b296-ef02c3fee70c\" (UID: \"fe048780-6d3f-4ba5-b296-ef02c3fee70c\") " Jul 27 16:24:44.299392 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.299364 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55fb6fb-63db-4382-bb60-1ac2a9bc7a14-bundle" (OuterVolumeSpecName: "bundle") pod "f55fb6fb-63db-4382-bb60-1ac2a9bc7a14" (UID: "f55fb6fb-63db-4382-bb60-1ac2a9bc7a14"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:24:44.299922 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.299880 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe048780-6d3f-4ba5-b296-ef02c3fee70c-bundle" (OuterVolumeSpecName: "bundle") pod "fe048780-6d3f-4ba5-b296-ef02c3fee70c" (UID: "fe048780-6d3f-4ba5-b296-ef02c3fee70c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:24:44.301145 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.301118 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55fb6fb-63db-4382-bb60-1ac2a9bc7a14-kube-api-access-g8zbs" (OuterVolumeSpecName: "kube-api-access-g8zbs") pod "f55fb6fb-63db-4382-bb60-1ac2a9bc7a14" (UID: "f55fb6fb-63db-4382-bb60-1ac2a9bc7a14"). InnerVolumeSpecName "kube-api-access-g8zbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:24:44.301420 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.301393 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe048780-6d3f-4ba5-b296-ef02c3fee70c-kube-api-access-gklp9" (OuterVolumeSpecName: "kube-api-access-gklp9") pod "fe048780-6d3f-4ba5-b296-ef02c3fee70c" (UID: "fe048780-6d3f-4ba5-b296-ef02c3fee70c"). InnerVolumeSpecName "kube-api-access-gklp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:24:44.304212 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.304186 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55fb6fb-63db-4382-bb60-1ac2a9bc7a14-util" (OuterVolumeSpecName: "util") pod "f55fb6fb-63db-4382-bb60-1ac2a9bc7a14" (UID: "f55fb6fb-63db-4382-bb60-1ac2a9bc7a14"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:24:44.306758 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.306711 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe048780-6d3f-4ba5-b296-ef02c3fee70c-util" (OuterVolumeSpecName: "util") pod "fe048780-6d3f-4ba5-b296-ef02c3fee70c" (UID: "fe048780-6d3f-4ba5-b296-ef02c3fee70c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:24:44.400093 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.399999 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe048780-6d3f-4ba5-b296-ef02c3fee70c-bundle\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:24:44.400093 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.400034 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f55fb6fb-63db-4382-bb60-1ac2a9bc7a14-util\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:24:44.400093 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.400042 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8zbs\" (UniqueName: \"kubernetes.io/projected/f55fb6fb-63db-4382-bb60-1ac2a9bc7a14-kube-api-access-g8zbs\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:24:44.400093 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.400052 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f55fb6fb-63db-4382-bb60-1ac2a9bc7a14-bundle\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:24:44.400093 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.400060 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe048780-6d3f-4ba5-b296-ef02c3fee70c-util\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:24:44.400093 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:44.400068 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gklp9\" (UniqueName: \"kubernetes.io/projected/fe048780-6d3f-4ba5-b296-ef02c3fee70c-kube-api-access-gklp9\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:24:45.098982 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:45.098937 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz" event={"ID":"fe048780-6d3f-4ba5-b296-ef02c3fee70c","Type":"ContainerDied","Data":"f1aed7fd09045088dce4068793966463b2afd799b8f33415a5b4aedeaed28a8b"} Jul 27 16:24:45.098982 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:45.098966 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz" Jul 27 16:24:45.099213 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:45.098991 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1aed7fd09045088dce4068793966463b2afd799b8f33415a5b4aedeaed28a8b" Jul 27 16:24:45.101026 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:45.100997 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" Jul 27 16:24:45.101172 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:45.100997 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p" event={"ID":"f55fb6fb-63db-4382-bb60-1ac2a9bc7a14","Type":"ContainerDied","Data":"c0feac8c7d1f8be757225d8caf6422fd3c8c4a6f76b1a1331de654cad9cde917"} Jul 27 16:24:45.101172 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:45.101112 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0feac8c7d1f8be757225d8caf6422fd3c8c4a6f76b1a1331de654cad9cde917" Jul 27 16:24:52.190697 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.190663 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kj6pt"] Jul 27 16:24:52.191085 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191070 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe048780-6d3f-4ba5-b296-ef02c3fee70c" containerName="util" Jul 27 16:24:52.191085 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191082 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe048780-6d3f-4ba5-b296-ef02c3fee70c" containerName="util" Jul 27 16:24:52.191177 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191091 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25a355db-00df-4c8f-82f1-13a486427ea3" containerName="pull" Jul 27 16:24:52.191177 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191097 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="25a355db-00df-4c8f-82f1-13a486427ea3" containerName="pull" Jul 27 16:24:52.191177 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191108 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6fd183d-09c5-43b3-894e-99a5c0c4c6d4" containerName="util" Jul 27 16:24:52.191177 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191113 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6fd183d-09c5-43b3-894e-99a5c0c4c6d4" containerName="util" Jul 27 16:24:52.191177 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191123 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25a355db-00df-4c8f-82f1-13a486427ea3" containerName="util" Jul 27 16:24:52.191177 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191128 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="25a355db-00df-4c8f-82f1-13a486427ea3" containerName="util" Jul 27 16:24:52.191177 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191135 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe048780-6d3f-4ba5-b296-ef02c3fee70c" containerName="extract" Jul 27 16:24:52.191177 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191140 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe048780-6d3f-4ba5-b296-ef02c3fee70c" containerName="extract" Jul 27 16:24:52.191177 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191148 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25a355db-00df-4c8f-82f1-13a486427ea3" containerName="extract" Jul 27 16:24:52.191177 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191153 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="25a355db-00df-4c8f-82f1-13a486427ea3" containerName="extract" Jul 27 16:24:52.191177 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191159 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe048780-6d3f-4ba5-b296-ef02c3fee70c" containerName="pull" Jul 27 16:24:52.191177 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191163 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe048780-6d3f-4ba5-b296-ef02c3fee70c" containerName="pull" Jul 27 16:24:52.191177 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191168 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55fb6fb-63db-4382-bb60-1ac2a9bc7a14" containerName="extract" Jul 27 16:24:52.191177 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191172 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55fb6fb-63db-4382-bb60-1ac2a9bc7a14" containerName="extract" Jul 27 16:24:52.191560 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191185 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55fb6fb-63db-4382-bb60-1ac2a9bc7a14" containerName="util" Jul 27 16:24:52.191560 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191189 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55fb6fb-63db-4382-bb60-1ac2a9bc7a14" containerName="util" Jul 27 16:24:52.191560 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191196 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6fd183d-09c5-43b3-894e-99a5c0c4c6d4" containerName="pull" Jul 27 16:24:52.191560 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191201 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6fd183d-09c5-43b3-894e-99a5c0c4c6d4" containerName="pull" Jul 27 16:24:52.191560 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191207 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6fd183d-09c5-43b3-894e-99a5c0c4c6d4" containerName="extract" Jul 27 16:24:52.191560 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191211 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6fd183d-09c5-43b3-894e-99a5c0c4c6d4" containerName="extract" Jul 27 16:24:52.191560 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191217 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55fb6fb-63db-4382-bb60-1ac2a9bc7a14" containerName="pull" Jul 27 16:24:52.191560 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191221 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55fb6fb-63db-4382-bb60-1ac2a9bc7a14" containerName="pull" Jul 27 16:24:52.191560 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191272 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="25a355db-00df-4c8f-82f1-13a486427ea3" containerName="extract" Jul 27 16:24:52.191560 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191280 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f55fb6fb-63db-4382-bb60-1ac2a9bc7a14" containerName="extract" Jul 27 16:24:52.191560 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191286 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe048780-6d3f-4ba5-b296-ef02c3fee70c" containerName="extract" Jul 27 16:24:52.191560 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.191294 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6fd183d-09c5-43b3-894e-99a5c0c4c6d4" containerName="extract" Jul 27 16:24:52.194592 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.194575 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kj6pt" Jul 27 16:24:52.198294 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.198272 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jtgxn\"" Jul 27 16:24:52.212814 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.212787 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kj6pt"] Jul 27 16:24:52.269475 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.269432 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcx97\" (UniqueName: \"kubernetes.io/projected/a236c9ad-7456-4935-bc28-9dc31aeaf99a-kube-api-access-bcx97\") pod \"authorino-operator-75cc8d5cb7-kj6pt\" (UID: \"a236c9ad-7456-4935-bc28-9dc31aeaf99a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kj6pt" Jul 27 16:24:52.370528 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.370484 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcx97\" (UniqueName: \"kubernetes.io/projected/a236c9ad-7456-4935-bc28-9dc31aeaf99a-kube-api-access-bcx97\") pod \"authorino-operator-75cc8d5cb7-kj6pt\" (UID: \"a236c9ad-7456-4935-bc28-9dc31aeaf99a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kj6pt" Jul 27 16:24:52.380974 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.380936 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcx97\" (UniqueName: \"kubernetes.io/projected/a236c9ad-7456-4935-bc28-9dc31aeaf99a-kube-api-access-bcx97\") pod \"authorino-operator-75cc8d5cb7-kj6pt\" (UID: \"a236c9ad-7456-4935-bc28-9dc31aeaf99a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kj6pt" Jul 27 16:24:52.505325 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.505232 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kj6pt" Jul 27 16:24:52.654139 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.654115 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78dc97c9f7-d6mqz"] Jul 27 16:24:52.655580 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:24:52.655550 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda236c9ad_7456_4935_bc28_9dc31aeaf99a.slice/crio-232b7ed4a6667dfa23cacf97bb664bb3ca18b74a0dfb5c63d82748c6f3fc8037 WatchSource:0}: Error finding container 232b7ed4a6667dfa23cacf97bb664bb3ca18b74a0dfb5c63d82748c6f3fc8037: Status 404 returned error can't find the container with id 232b7ed4a6667dfa23cacf97bb664bb3ca18b74a0dfb5c63d82748c6f3fc8037 Jul 27 16:24:52.658944 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.658924 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kj6pt"] Jul 27 16:24:52.659041 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.659022 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.690969 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.679990 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78dc97c9f7-d6mqz"] Jul 27 16:24:52.781194 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.781156 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19a2f7be-7dea-4225-83b0-3aea575091bc-console-config\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.781373 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.781205 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19a2f7be-7dea-4225-83b0-3aea575091bc-service-ca\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.781373 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.781264 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19a2f7be-7dea-4225-83b0-3aea575091bc-console-serving-cert\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.781373 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.781329 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57qm2\" (UniqueName: \"kubernetes.io/projected/19a2f7be-7dea-4225-83b0-3aea575091bc-kube-api-access-57qm2\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.781373 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.781356 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19a2f7be-7dea-4225-83b0-3aea575091bc-trusted-ca-bundle\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.781551 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.781396 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19a2f7be-7dea-4225-83b0-3aea575091bc-oauth-serving-cert\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.781551 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.781423 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19a2f7be-7dea-4225-83b0-3aea575091bc-console-oauth-config\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.881933 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.881876 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19a2f7be-7dea-4225-83b0-3aea575091bc-oauth-serving-cert\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.882090 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.881968 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19a2f7be-7dea-4225-83b0-3aea575091bc-console-oauth-config\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.882090 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.882016 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19a2f7be-7dea-4225-83b0-3aea575091bc-console-config\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.882090 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.882051 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19a2f7be-7dea-4225-83b0-3aea575091bc-service-ca\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.882363 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.882088 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19a2f7be-7dea-4225-83b0-3aea575091bc-console-serving-cert\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.882363 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.882137 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57qm2\" (UniqueName: \"kubernetes.io/projected/19a2f7be-7dea-4225-83b0-3aea575091bc-kube-api-access-57qm2\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.882363 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.882346 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19a2f7be-7dea-4225-83b0-3aea575091bc-trusted-ca-bundle\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.882731 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.882704 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19a2f7be-7dea-4225-83b0-3aea575091bc-oauth-serving-cert\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.882883 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.882832 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19a2f7be-7dea-4225-83b0-3aea575091bc-service-ca\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.882883 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.882960 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19a2f7be-7dea-4225-83b0-3aea575091bc-console-config\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.883153 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.883113 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19a2f7be-7dea-4225-83b0-3aea575091bc-trusted-ca-bundle\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.884625 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.884601 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19a2f7be-7dea-4225-83b0-3aea575091bc-console-oauth-config\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.884714 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.884681 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19a2f7be-7dea-4225-83b0-3aea575091bc-console-serving-cert\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.905378 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.905353 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57qm2\" (UniqueName: \"kubernetes.io/projected/19a2f7be-7dea-4225-83b0-3aea575091bc-kube-api-access-57qm2\") pod \"console-78dc97c9f7-d6mqz\" (UID: \"19a2f7be-7dea-4225-83b0-3aea575091bc\") " pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:52.983897 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:52.983858 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:24:53.131755 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:53.131725 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78dc97c9f7-d6mqz"] Jul 27 16:24:53.132806 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:24:53.132777 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a2f7be_7dea_4225_83b0_3aea575091bc.slice/crio-4deb482172ea79832c263d82d5d39e1bc369ee0b7b04bd3337b2efef59c5f3a3 WatchSource:0}: Error finding container 4deb482172ea79832c263d82d5d39e1bc369ee0b7b04bd3337b2efef59c5f3a3: Status 404 returned error can't find the container with id 4deb482172ea79832c263d82d5d39e1bc369ee0b7b04bd3337b2efef59c5f3a3 Jul 27 16:24:53.144087 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:53.144060 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kj6pt" event={"ID":"a236c9ad-7456-4935-bc28-9dc31aeaf99a","Type":"ContainerStarted","Data":"232b7ed4a6667dfa23cacf97bb664bb3ca18b74a0dfb5c63d82748c6f3fc8037"} Jul 27 16:24:53.145506 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:53.145483 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78dc97c9f7-d6mqz" event={"ID":"19a2f7be-7dea-4225-83b0-3aea575091bc","Type":"ContainerStarted","Data":"4deb482172ea79832c263d82d5d39e1bc369ee0b7b04bd3337b2efef59c5f3a3"} Jul 27 16:24:54.155240 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:54.154714 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78dc97c9f7-d6mqz" event={"ID":"19a2f7be-7dea-4225-83b0-3aea575091bc","Type":"ContainerStarted","Data":"4f418fe7ca48603ca8184d5e1d1ad4603e0946d4d8b66260daa423276dacd2cf"} Jul 27 16:24:54.187807 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:54.187751 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78dc97c9f7-d6mqz" podStartSLOduration=2.18773441 podStartE2EDuration="2.18773441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:24:54.184581217 +0000 UTC m=+472.857115717" watchObservedRunningTime="2026-07-27 16:24:54.18773441 +0000 UTC m=+472.860268909" Jul 27 16:24:55.162416 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:55.162379 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kj6pt" event={"ID":"a236c9ad-7456-4935-bc28-9dc31aeaf99a","Type":"ContainerStarted","Data":"633a42fc9e7575f7674ad53da286b8356d5f04b92b3986828e2c82a2d4d59188"} Jul 27 16:24:55.162812 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:55.162454 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kj6pt" Jul 27 16:24:55.184776 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:55.184720 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kj6pt" podStartSLOduration=0.78927205 podStartE2EDuration="3.184703438s" totalImagesPullingTime="2.395431388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:24:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:24:55.184123561 +0000 UTC m=+473.856658061" watchObservedRunningTime="2026-07-27 16:24:55.184703438 +0000 UTC m=+473.857237937" Jul 27 16:24:55.930898 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:55.930861 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622"] Jul 27 16:24:55.934631 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:55.934611 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" Jul 27 16:24:55.940371 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:55.940347 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zjx4c\"" Jul 27 16:24:55.960420 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:55.960386 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622"] Jul 27 16:24:56.015040 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:56.015001 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bfml\" (UniqueName: \"kubernetes.io/projected/2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4-kube-api-access-4bfml\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8r622\" (UID: \"2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" Jul 27 16:24:56.015203 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:56.015134 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8r622\" (UID: \"2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" Jul 27 16:24:56.115962 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:56.115886 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bfml\" (UniqueName: \"kubernetes.io/projected/2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4-kube-api-access-4bfml\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8r622\" (UID: \"2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" Jul 27 16:24:56.116148 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:56.116010 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8r622\" (UID: \"2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" Jul 27 16:24:56.116347 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:56.116331 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8r622\" (UID: \"2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" Jul 27 16:24:56.134109 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:56.134079 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bfml\" (UniqueName: \"kubernetes.io/projected/2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4-kube-api-access-4bfml\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8r622\" (UID: \"2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" Jul 27 16:24:56.246173 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:56.246084 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" Jul 27 16:24:56.395641 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:56.395612 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622"] Jul 27 16:24:56.397665 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:24:56.397637 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2602e1dc_efb2_4d2d_9f00_5c2a6dcc00f4.slice/crio-57e19eea19be5ce4b68e8556637692810a66aba4abeaa211d252f02d5275d7c3 WatchSource:0}: Error finding container 57e19eea19be5ce4b68e8556637692810a66aba4abeaa211d252f02d5275d7c3: Status 404 returned error can't find the container with id 57e19eea19be5ce4b68e8556637692810a66aba4abeaa211d252f02d5275d7c3 Jul 27 16:24:57.176578 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:57.176539 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" event={"ID":"2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4","Type":"ContainerStarted","Data":"57e19eea19be5ce4b68e8556637692810a66aba4abeaa211d252f02d5275d7c3"} Jul 27 16:24:58.013632 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:58.013590 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-njb5q"] Jul 27 16:24:58.017567 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:58.017539 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njb5q" Jul 27 16:24:58.021841 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:58.021818 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 27 16:24:58.022148 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:58.022135 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-s7htc\"" Jul 27 16:24:58.032575 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:58.032554 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-njb5q"] Jul 27 16:24:58.134301 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:58.134264 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm8mx\" (UniqueName: \"kubernetes.io/projected/feaa1fea-b20f-41e4-b290-35f7f9b7bdda-kube-api-access-sm8mx\") pod \"dns-operator-controller-manager-864dd78c4c-njb5q\" (UID: \"feaa1fea-b20f-41e4-b290-35f7f9b7bdda\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njb5q" Jul 27 16:24:58.235728 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:58.235680 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm8mx\" (UniqueName: \"kubernetes.io/projected/feaa1fea-b20f-41e4-b290-35f7f9b7bdda-kube-api-access-sm8mx\") pod \"dns-operator-controller-manager-864dd78c4c-njb5q\" (UID: \"feaa1fea-b20f-41e4-b290-35f7f9b7bdda\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njb5q" Jul 27 16:24:58.250384 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:58.250352 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm8mx\" (UniqueName: \"kubernetes.io/projected/feaa1fea-b20f-41e4-b290-35f7f9b7bdda-kube-api-access-sm8mx\") pod \"dns-operator-controller-manager-864dd78c4c-njb5q\" (UID: \"feaa1fea-b20f-41e4-b290-35f7f9b7bdda\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njb5q" Jul 27 16:24:58.328826 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:58.328742 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njb5q" Jul 27 16:24:58.502917 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:58.502877 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-njb5q"] Jul 27 16:24:58.504623 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:24:58.504587 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeaa1fea_b20f_41e4_b290_35f7f9b7bdda.slice/crio-af0cc711376f276210286ce27f803a14c4fd156f031dde93ee05acbe7bb4818c WatchSource:0}: Error finding container af0cc711376f276210286ce27f803a14c4fd156f031dde93ee05acbe7bb4818c: Status 404 returned error can't find the container with id af0cc711376f276210286ce27f803a14c4fd156f031dde93ee05acbe7bb4818c Jul 27 16:24:59.189443 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:24:59.189409 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njb5q" event={"ID":"feaa1fea-b20f-41e4-b290-35f7f9b7bdda","Type":"ContainerStarted","Data":"af0cc711376f276210286ce27f803a14c4fd156f031dde93ee05acbe7bb4818c"} Jul 27 16:25:00.564253 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:00.564203 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj"] Jul 27 16:25:00.568698 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:00.568669 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj" Jul 27 16:25:00.572439 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:00.572383 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kntjl\"" Jul 27 16:25:00.576774 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:00.576747 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj"] Jul 27 16:25:00.663505 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:00.663467 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-295w4\" (UniqueName: \"kubernetes.io/projected/c052b62f-7572-492a-8c7c-79ed44ee2c87-kube-api-access-295w4\") pod \"limitador-operator-controller-manager-859d7b78bd-jpqzj\" (UID: \"c052b62f-7572-492a-8c7c-79ed44ee2c87\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj" Jul 27 16:25:00.764377 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:00.764345 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-295w4\" (UniqueName: \"kubernetes.io/projected/c052b62f-7572-492a-8c7c-79ed44ee2c87-kube-api-access-295w4\") pod \"limitador-operator-controller-manager-859d7b78bd-jpqzj\" (UID: \"c052b62f-7572-492a-8c7c-79ed44ee2c87\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj" Jul 27 16:25:00.773977 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:00.773952 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-295w4\" (UniqueName: \"kubernetes.io/projected/c052b62f-7572-492a-8c7c-79ed44ee2c87-kube-api-access-295w4\") pod \"limitador-operator-controller-manager-859d7b78bd-jpqzj\" (UID: \"c052b62f-7572-492a-8c7c-79ed44ee2c87\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj" Jul 27 16:25:00.883951 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:00.883849 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj" Jul 27 16:25:01.055497 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:01.055466 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj"] Jul 27 16:25:01.057144 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:25:01.057118 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc052b62f_7572_492a_8c7c_79ed44ee2c87.slice/crio-c88d124f93ffd15f2afe3c363246256567ebc55d4cc71ac1db1a6dc8468435bf WatchSource:0}: Error finding container c88d124f93ffd15f2afe3c363246256567ebc55d4cc71ac1db1a6dc8468435bf: Status 404 returned error can't find the container with id c88d124f93ffd15f2afe3c363246256567ebc55d4cc71ac1db1a6dc8468435bf Jul 27 16:25:01.204552 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:01.204462 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj" event={"ID":"c052b62f-7572-492a-8c7c-79ed44ee2c87","Type":"ContainerStarted","Data":"c88d124f93ffd15f2afe3c363246256567ebc55d4cc71ac1db1a6dc8468435bf"} Jul 27 16:25:01.206083 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:01.206056 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njb5q" event={"ID":"feaa1fea-b20f-41e4-b290-35f7f9b7bdda","Type":"ContainerStarted","Data":"2cd54bed694a66573073a87dfc51ba37142f5f8e780b019ba8ea8815b0416aa5"} Jul 27 16:25:01.206196 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:01.206115 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njb5q" Jul 27 16:25:01.234058 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:01.233996 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njb5q" podStartSLOduration=2.069161054 podStartE2EDuration="4.233977471s" totalImagesPullingTime="2.164816417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:25:01.23349855 +0000 UTC m=+479.906033048" watchObservedRunningTime="2026-07-27 16:25:01.233977471 +0000 UTC m=+479.906511970" Jul 27 16:25:02.984366 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:02.984334 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:25:02.984837 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:02.984395 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:25:02.990023 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:02.989999 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:25:03.223705 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:03.223675 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78dc97c9f7-d6mqz" Jul 27 16:25:03.305101 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:03.305065 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d5c4dfbcd-8h8ft"] Jul 27 16:25:04.227042 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:04.227005 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" event={"ID":"2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4","Type":"ContainerStarted","Data":"35944ea607085104207be82d284a44af4f91ab2aedfc82ec2a464914db6342c0"} Jul 27 16:25:04.227513 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:04.227125 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" Jul 27 16:25:04.228806 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:04.228783 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj" event={"ID":"c052b62f-7572-492a-8c7c-79ed44ee2c87","Type":"ContainerStarted","Data":"37b9971368a35bfd1710bb855a51e6d9878995a090213609b01ea05d93c3f8ad"} Jul 27 16:25:04.228963 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:04.228946 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj" Jul 27 16:25:04.251431 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:04.251380 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" podStartSLOduration=1.518861048 podStartE2EDuration="9.251363721s" totalImagesPullingTime="7.732502673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:25:04.248810389 +0000 UTC m=+482.921344901" watchObservedRunningTime="2026-07-27 16:25:04.251363721 +0000 UTC m=+482.923898220" Jul 27 16:25:04.271241 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:04.271185 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj" podStartSLOduration=1.204820247 podStartE2EDuration="4.271168413s" totalImagesPullingTime="3.066348166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:25:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:25:04.268723864 +0000 UTC m=+482.941258363" watchObservedRunningTime="2026-07-27 16:25:04.271168413 +0000 UTC m=+482.943702911" Jul 27 16:25:06.171263 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:06.171232 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kj6pt" Jul 27 16:25:07.042534 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:07.042499 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dhx4s"] Jul 27 16:25:07.046149 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:07.046127 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dhx4s" Jul 27 16:25:07.049315 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:07.049294 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 27 16:25:07.049652 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:07.049635 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 27 16:25:07.049652 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:07.049647 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2xkwb\"" Jul 27 16:25:07.059229 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:07.059205 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dhx4s"] Jul 27 16:25:07.129110 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:07.129073 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbqfw\" (UniqueName: \"kubernetes.io/projected/f67521c8-88be-43e7-9136-c7817310801f-kube-api-access-pbqfw\") pod \"kuadrant-console-plugin-66b5774985-dhx4s\" (UID: \"f67521c8-88be-43e7-9136-c7817310801f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dhx4s" Jul 27 16:25:07.129304 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:07.129128 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f67521c8-88be-43e7-9136-c7817310801f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dhx4s\" (UID: \"f67521c8-88be-43e7-9136-c7817310801f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dhx4s" Jul 27 16:25:07.129304 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:07.129193 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f67521c8-88be-43e7-9136-c7817310801f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dhx4s\" (UID: \"f67521c8-88be-43e7-9136-c7817310801f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dhx4s" Jul 27 16:25:07.230662 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:07.230634 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbqfw\" (UniqueName: \"kubernetes.io/projected/f67521c8-88be-43e7-9136-c7817310801f-kube-api-access-pbqfw\") pod \"kuadrant-console-plugin-66b5774985-dhx4s\" (UID: \"f67521c8-88be-43e7-9136-c7817310801f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dhx4s" Jul 27 16:25:07.231153 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:07.230671 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f67521c8-88be-43e7-9136-c7817310801f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dhx4s\" (UID: \"f67521c8-88be-43e7-9136-c7817310801f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dhx4s" Jul 27 16:25:07.231153 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:07.230704 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f67521c8-88be-43e7-9136-c7817310801f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dhx4s\" (UID: \"f67521c8-88be-43e7-9136-c7817310801f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dhx4s" Jul 27 16:25:07.231468 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:07.231446 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f67521c8-88be-43e7-9136-c7817310801f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dhx4s\" (UID: \"f67521c8-88be-43e7-9136-c7817310801f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dhx4s" Jul 27 16:25:07.233351 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:07.233334 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f67521c8-88be-43e7-9136-c7817310801f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dhx4s\" (UID: \"f67521c8-88be-43e7-9136-c7817310801f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dhx4s" Jul 27 16:25:07.250242 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:07.250214 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbqfw\" (UniqueName: \"kubernetes.io/projected/f67521c8-88be-43e7-9136-c7817310801f-kube-api-access-pbqfw\") pod \"kuadrant-console-plugin-66b5774985-dhx4s\" (UID: \"f67521c8-88be-43e7-9136-c7817310801f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dhx4s" Jul 27 16:25:07.358076 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:07.357990 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dhx4s" Jul 27 16:25:07.503036 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:07.503000 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dhx4s"] Jul 27 16:25:07.506116 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:25:07.506082 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf67521c8_88be_43e7_9136_c7817310801f.slice/crio-d52c034be788276660e495310a7c2d7f9db3c39917faa12e4feb844de98b2b01 WatchSource:0}: Error finding container d52c034be788276660e495310a7c2d7f9db3c39917faa12e4feb844de98b2b01: Status 404 returned error can't find the container with id d52c034be788276660e495310a7c2d7f9db3c39917faa12e4feb844de98b2b01 Jul 27 16:25:08.255476 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:08.255444 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dhx4s" event={"ID":"f67521c8-88be-43e7-9136-c7817310801f","Type":"ContainerStarted","Data":"d52c034be788276660e495310a7c2d7f9db3c39917faa12e4feb844de98b2b01"} Jul 27 16:25:12.216137 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:12.216109 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njb5q" Jul 27 16:25:15.237052 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:15.237021 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" Jul 27 16:25:15.237438 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:15.237100 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj" Jul 27 16:25:16.682116 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:16.682081 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d"] Jul 27 16:25:16.702982 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:16.702941 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d" Jul 27 16:25:16.711751 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:16.711714 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d"] Jul 27 16:25:16.826665 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:16.826635 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spp2q\" (UniqueName: \"kubernetes.io/projected/a5ce20e6-4303-4b07-a4e4-5844d49ab9d9-kube-api-access-spp2q\") pod \"kuadrant-operator-controller-manager-55467d9f7-v267d\" (UID: \"a5ce20e6-4303-4b07-a4e4-5844d49ab9d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d" Jul 27 16:25:16.826880 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:16.826687 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5ce20e6-4303-4b07-a4e4-5844d49ab9d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-v267d\" (UID: \"a5ce20e6-4303-4b07-a4e4-5844d49ab9d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d" Jul 27 16:25:16.927482 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:16.927441 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spp2q\" (UniqueName: \"kubernetes.io/projected/a5ce20e6-4303-4b07-a4e4-5844d49ab9d9-kube-api-access-spp2q\") pod \"kuadrant-operator-controller-manager-55467d9f7-v267d\" (UID: \"a5ce20e6-4303-4b07-a4e4-5844d49ab9d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d" Jul 27 16:25:16.927683 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:16.927539 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5ce20e6-4303-4b07-a4e4-5844d49ab9d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-v267d\" (UID: \"a5ce20e6-4303-4b07-a4e4-5844d49ab9d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d" Jul 27 16:25:16.928018 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:16.927976 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5ce20e6-4303-4b07-a4e4-5844d49ab9d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-v267d\" (UID: \"a5ce20e6-4303-4b07-a4e4-5844d49ab9d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d" Jul 27 16:25:16.950475 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:16.950405 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spp2q\" (UniqueName: \"kubernetes.io/projected/a5ce20e6-4303-4b07-a4e4-5844d49ab9d9-kube-api-access-spp2q\") pod \"kuadrant-operator-controller-manager-55467d9f7-v267d\" (UID: \"a5ce20e6-4303-4b07-a4e4-5844d49ab9d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d" Jul 27 16:25:17.023662 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.023612 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d" Jul 27 16:25:17.186867 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.186831 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d"] Jul 27 16:25:17.193943 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.193873 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d"] Jul 27 16:25:17.209773 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.209681 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622"] Jul 27 16:25:17.211944 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.209942 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" podUID="2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4" containerName="manager" containerID="cri-o://35944ea607085104207be82d284a44af4f91ab2aedfc82ec2a464914db6342c0" gracePeriod=2 Jul 27 16:25:17.215500 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.215470 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn"] Jul 27 16:25:17.219736 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.219677 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn" Jul 27 16:25:17.227647 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.227618 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622"] Jul 27 16:25:17.237937 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.237886 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn"] Jul 27 16:25:17.240528 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:25:17.240488 2550 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-8r622\": pods \"kuadrant-operator-controller-manager-8684c9ff49-8r622\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" Jul 27 16:25:17.245090 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:25:17.244986 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8r622\" is forbidden: User \"system:node:ip-10-0-135-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-70.ec2.internal' and this object" podUID="2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" Jul 27 16:25:17.245415 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.245390 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj"] Jul 27 16:25:17.245678 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.245548 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj" podUID="c052b62f-7572-492a-8c7c-79ed44ee2c87" containerName="manager" containerID="cri-o://37b9971368a35bfd1710bb855a51e6d9878995a090213609b01ea05d93c3f8ad" gracePeriod=2 Jul 27 16:25:17.257041 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.256997 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj"] Jul 27 16:25:17.274604 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:25:17.274482 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jpqzj\" is forbidden: User \"system:node:ip-10-0-135-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-70.ec2.internal' and this object" podUID="c052b62f-7572-492a-8c7c-79ed44ee2c87" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj" Jul 27 16:25:17.278948 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:25:17.276803 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8r622\" is forbidden: User \"system:node:ip-10-0-135-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-70.ec2.internal' and this object" podUID="2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" Jul 27 16:25:17.280434 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.280402 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mj5zs"] Jul 27 16:25:17.281137 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.281116 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c052b62f-7572-492a-8c7c-79ed44ee2c87" containerName="manager" Jul 27 16:25:17.281226 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.281140 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c052b62f-7572-492a-8c7c-79ed44ee2c87" containerName="manager" Jul 27 16:25:17.281226 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.281154 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4" containerName="manager" Jul 27 16:25:17.281226 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.281162 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4" containerName="manager" Jul 27 16:25:17.281324 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.281262 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4" containerName="manager" Jul 27 16:25:17.281324 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.281278 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c052b62f-7572-492a-8c7c-79ed44ee2c87" containerName="manager" Jul 27 16:25:17.284956 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.284837 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mj5zs" Jul 27 16:25:17.287856 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:25:17.287704 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jpqzj\" is forbidden: User \"system:node:ip-10-0-135-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-70.ec2.internal' and this object" podUID="c052b62f-7572-492a-8c7c-79ed44ee2c87" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj" Jul 27 16:25:17.290270 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:25:17.290210 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8r622\" is forbidden: User \"system:node:ip-10-0-135-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-70.ec2.internal' and this object" podUID="2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" Jul 27 16:25:17.293576 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.293550 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mj5zs"] Jul 27 16:25:17.334458 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.334378 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fl2n\" (UniqueName: \"kubernetes.io/projected/83affb8e-8575-4e39-875b-7bd4990a87e3-kube-api-access-5fl2n\") pod \"limitador-operator-controller-manager-859d7b78bd-mj5zs\" (UID: \"83affb8e-8575-4e39-875b-7bd4990a87e3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mj5zs" Jul 27 16:25:17.334633 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.334504 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/219d559f-a077-4984-92e3-bc87c11610b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-jxlzn\" (UID: \"219d559f-a077-4984-92e3-bc87c11610b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn" Jul 27 16:25:17.334692 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.334640 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tp9b\" (UniqueName: \"kubernetes.io/projected/219d559f-a077-4984-92e3-bc87c11610b8-kube-api-access-4tp9b\") pod \"kuadrant-operator-controller-manager-55467d9f7-jxlzn\" (UID: \"219d559f-a077-4984-92e3-bc87c11610b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn" Jul 27 16:25:17.436332 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.436190 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tp9b\" (UniqueName: \"kubernetes.io/projected/219d559f-a077-4984-92e3-bc87c11610b8-kube-api-access-4tp9b\") pod \"kuadrant-operator-controller-manager-55467d9f7-jxlzn\" (UID: \"219d559f-a077-4984-92e3-bc87c11610b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn" Jul 27 16:25:17.436332 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.436242 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fl2n\" (UniqueName: \"kubernetes.io/projected/83affb8e-8575-4e39-875b-7bd4990a87e3-kube-api-access-5fl2n\") pod \"limitador-operator-controller-manager-859d7b78bd-mj5zs\" (UID: \"83affb8e-8575-4e39-875b-7bd4990a87e3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mj5zs" Jul 27 16:25:17.436485 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.436331 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/219d559f-a077-4984-92e3-bc87c11610b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-jxlzn\" (UID: \"219d559f-a077-4984-92e3-bc87c11610b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn" Jul 27 16:25:17.437455 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.437402 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/219d559f-a077-4984-92e3-bc87c11610b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-jxlzn\" (UID: \"219d559f-a077-4984-92e3-bc87c11610b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn" Jul 27 16:25:17.455262 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.455229 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tp9b\" (UniqueName: \"kubernetes.io/projected/219d559f-a077-4984-92e3-bc87c11610b8-kube-api-access-4tp9b\") pod \"kuadrant-operator-controller-manager-55467d9f7-jxlzn\" (UID: \"219d559f-a077-4984-92e3-bc87c11610b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn" Jul 27 16:25:17.455430 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.455226 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fl2n\" (UniqueName: \"kubernetes.io/projected/83affb8e-8575-4e39-875b-7bd4990a87e3-kube-api-access-5fl2n\") pod \"limitador-operator-controller-manager-859d7b78bd-mj5zs\" (UID: \"83affb8e-8575-4e39-875b-7bd4990a87e3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mj5zs" Jul 27 16:25:17.464415 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:25:17.464362 2550 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 27 16:25:17.464415 ip-10-0-135-70 kubenswrapper[2550]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-v267d_kuadrant-system_a5ce20e6-4303-4b07-a4e4-5844d49ab9d9_0(68ac190f7ca30fa98db735a47ed8d671eeaab654b754a5a1b97d80b4dec76b99): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-v267d to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"68ac190f7ca30fa98db735a47ed8d671eeaab654b754a5a1b97d80b4dec76b99" Netns:"/var/run/netns/521a0e1c-227e-4238-afc5-338860296caf" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-v267d;K8S_POD_INFRA_CONTAINER_ID=68ac190f7ca30fa98db735a47ed8d671eeaab654b754a5a1b97d80b4dec76b99;K8S_POD_UID=a5ce20e6-4303-4b07-a4e4-5844d49ab9d9" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d] networking: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d/a5ce20e6-4303-4b07-a4e4-5844d49ab9d9:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d 68ac190f7ca30fa98db735a47ed8d671eeaab654b754a5a1b97d80b4dec76b99 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:85:00:32 [10.133.0.50/23] Jul 27 16:25:17.464415 ip-10-0-135-70 kubenswrapper[2550]: ' Jul 27 16:25:17.464415 ip-10-0-135-70 kubenswrapper[2550]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 27 16:25:17.464415 ip-10-0-135-70 kubenswrapper[2550]: > Jul 27 16:25:17.464694 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:25:17.464452 2550 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 27 16:25:17.464694 ip-10-0-135-70 kubenswrapper[2550]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-v267d_kuadrant-system_a5ce20e6-4303-4b07-a4e4-5844d49ab9d9_0(68ac190f7ca30fa98db735a47ed8d671eeaab654b754a5a1b97d80b4dec76b99): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-v267d to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"68ac190f7ca30fa98db735a47ed8d671eeaab654b754a5a1b97d80b4dec76b99" Netns:"/var/run/netns/521a0e1c-227e-4238-afc5-338860296caf" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-v267d;K8S_POD_INFRA_CONTAINER_ID=68ac190f7ca30fa98db735a47ed8d671eeaab654b754a5a1b97d80b4dec76b99;K8S_POD_UID=a5ce20e6-4303-4b07-a4e4-5844d49ab9d9" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d] networking: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d/a5ce20e6-4303-4b07-a4e4-5844d49ab9d9:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d 68ac190f7ca30fa98db735a47ed8d671eeaab654b754a5a1b97d80b4dec76b99 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:85:00:32 [10.133.0.50/23] Jul 27 16:25:17.464694 ip-10-0-135-70 kubenswrapper[2550]: ' Jul 27 16:25:17.464694 ip-10-0-135-70 kubenswrapper[2550]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 27 16:25:17.464694 ip-10-0-135-70 kubenswrapper[2550]: > pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d" Jul 27 16:25:17.496973 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.496940 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" Jul 27 16:25:17.499488 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:25:17.499451 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jpqzj\" is forbidden: User \"system:node:ip-10-0-135-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-70.ec2.internal' and this object" podUID="c052b62f-7572-492a-8c7c-79ed44ee2c87" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj" Jul 27 16:25:17.501923 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:25:17.501877 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8r622\" is forbidden: User \"system:node:ip-10-0-135-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-70.ec2.internal' and this object" podUID="2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" Jul 27 16:25:17.507400 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.507360 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj" Jul 27 16:25:17.510006 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:25:17.509977 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jpqzj\" is forbidden: User \"system:node:ip-10-0-135-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-70.ec2.internal' and this object" podUID="c052b62f-7572-492a-8c7c-79ed44ee2c87" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj" Jul 27 16:25:17.512510 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:25:17.512470 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8r622\" is forbidden: User \"system:node:ip-10-0-135-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-70.ec2.internal' and this object" podUID="2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" Jul 27 16:25:17.537301 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.537267 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bfml\" (UniqueName: \"kubernetes.io/projected/2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4-kube-api-access-4bfml\") pod \"2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4\" (UID: \"2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4\") " Jul 27 16:25:17.537491 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.537336 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-295w4\" (UniqueName: \"kubernetes.io/projected/c052b62f-7572-492a-8c7c-79ed44ee2c87-kube-api-access-295w4\") pod \"c052b62f-7572-492a-8c7c-79ed44ee2c87\" (UID: \"c052b62f-7572-492a-8c7c-79ed44ee2c87\") " Jul 27 16:25:17.537491 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.537456 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4-extensions-socket-volume\") pod \"2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4\" (UID: \"2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4\") " Jul 27 16:25:17.538286 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.538253 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4" (UID: "2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:25:17.539642 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.539618 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4-kube-api-access-4bfml" (OuterVolumeSpecName: "kube-api-access-4bfml") pod "2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4" (UID: "2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4"). InnerVolumeSpecName "kube-api-access-4bfml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:25:17.539792 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.539771 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c052b62f-7572-492a-8c7c-79ed44ee2c87-kube-api-access-295w4" (OuterVolumeSpecName: "kube-api-access-295w4") pod "c052b62f-7572-492a-8c7c-79ed44ee2c87" (UID: "c052b62f-7572-492a-8c7c-79ed44ee2c87"). InnerVolumeSpecName "kube-api-access-295w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:25:17.633867 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.633831 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn" Jul 27 16:25:17.638333 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.638239 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bfml\" (UniqueName: \"kubernetes.io/projected/2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4-kube-api-access-4bfml\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:25:17.638333 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.638275 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-295w4\" (UniqueName: \"kubernetes.io/projected/c052b62f-7572-492a-8c7c-79ed44ee2c87-kube-api-access-295w4\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:25:17.638333 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.638293 2550 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4-extensions-socket-volume\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:25:17.642069 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.642036 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mj5zs" Jul 27 16:25:17.815356 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.815330 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn"] Jul 27 16:25:17.817669 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:25:17.817640 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219d559f_a077_4984_92e3_bc87c11610b8.slice/crio-a968e0762c319e554cfb75b624de4dc1c694b09f1a6bfdc7d96283d0bfde5115 WatchSource:0}: Error finding container a968e0762c319e554cfb75b624de4dc1c694b09f1a6bfdc7d96283d0bfde5115: Status 404 returned error can't find the container with id a968e0762c319e554cfb75b624de4dc1c694b09f1a6bfdc7d96283d0bfde5115 Jul 27 16:25:17.843414 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.843385 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mj5zs"] Jul 27 16:25:17.846194 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:25:17.846169 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83affb8e_8575_4e39_875b_7bd4990a87e3.slice/crio-7f39dec9aa95f91fba79807683c9508883e8ae7a01b518d17131e37300f4987e WatchSource:0}: Error finding container 7f39dec9aa95f91fba79807683c9508883e8ae7a01b518d17131e37300f4987e: Status 404 returned error can't find the container with id 7f39dec9aa95f91fba79807683c9508883e8ae7a01b518d17131e37300f4987e Jul 27 16:25:17.856641 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.856607 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4" path="/var/lib/kubelet/pods/2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4/volumes" Jul 27 16:25:17.857668 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:17.857645 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c052b62f-7572-492a-8c7c-79ed44ee2c87" path="/var/lib/kubelet/pods/c052b62f-7572-492a-8c7c-79ed44ee2c87/volumes" Jul 27 16:25:18.338485 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.338446 2550 generic.go:356] "Generic (PLEG): container finished" podID="c052b62f-7572-492a-8c7c-79ed44ee2c87" containerID="37b9971368a35bfd1710bb855a51e6d9878995a090213609b01ea05d93c3f8ad" exitCode=0 Jul 27 16:25:18.338687 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.338657 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jpqzj" Jul 27 16:25:18.339053 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.339037 2550 scope.go:117] "RemoveContainer" containerID="37b9971368a35bfd1710bb855a51e6d9878995a090213609b01ea05d93c3f8ad" Jul 27 16:25:18.342130 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.342102 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mj5zs" event={"ID":"83affb8e-8575-4e39-875b-7bd4990a87e3","Type":"ContainerStarted","Data":"24fc78269666855565bf920db618326cef2e8d400d7a5039e1b6a325d66477be"} Jul 27 16:25:18.342242 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.342147 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mj5zs" event={"ID":"83affb8e-8575-4e39-875b-7bd4990a87e3","Type":"ContainerStarted","Data":"7f39dec9aa95f91fba79807683c9508883e8ae7a01b518d17131e37300f4987e"} Jul 27 16:25:18.342242 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.342183 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mj5zs" Jul 27 16:25:18.344865 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.344455 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn" event={"ID":"219d559f-a077-4984-92e3-bc87c11610b8","Type":"ContainerStarted","Data":"e85fe0aa615fa41ad1407401169970aecff2b8f675a46744bba6bc0fb7e5e82f"} Jul 27 16:25:18.344865 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.344499 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn" event={"ID":"219d559f-a077-4984-92e3-bc87c11610b8","Type":"ContainerStarted","Data":"a968e0762c319e554cfb75b624de4dc1c694b09f1a6bfdc7d96283d0bfde5115"} Jul 27 16:25:18.344865 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.344519 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn" Jul 27 16:25:18.346434 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.346410 2550 generic.go:356] "Generic (PLEG): container finished" podID="2602e1dc-efb2-4d2d-9f00-5c2a6dcc00f4" containerID="35944ea607085104207be82d284a44af4f91ab2aedfc82ec2a464914db6342c0" exitCode=0 Jul 27 16:25:18.346532 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.346458 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8r622" Jul 27 16:25:18.346598 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.346572 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d" Jul 27 16:25:18.351275 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.351250 2550 scope.go:117] "RemoveContainer" containerID="37b9971368a35bfd1710bb855a51e6d9878995a090213609b01ea05d93c3f8ad" Jul 27 16:25:18.351617 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:25:18.351589 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b9971368a35bfd1710bb855a51e6d9878995a090213609b01ea05d93c3f8ad\": container with ID starting with 37b9971368a35bfd1710bb855a51e6d9878995a090213609b01ea05d93c3f8ad not found: ID does not exist" containerID="37b9971368a35bfd1710bb855a51e6d9878995a090213609b01ea05d93c3f8ad" Jul 27 16:25:18.351699 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.351630 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b9971368a35bfd1710bb855a51e6d9878995a090213609b01ea05d93c3f8ad"} err="failed to get container status \"37b9971368a35bfd1710bb855a51e6d9878995a090213609b01ea05d93c3f8ad\": rpc error: code = NotFound desc = could not find container \"37b9971368a35bfd1710bb855a51e6d9878995a090213609b01ea05d93c3f8ad\": container with ID starting with 37b9971368a35bfd1710bb855a51e6d9878995a090213609b01ea05d93c3f8ad not found: ID does not exist" Jul 27 16:25:18.351699 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.351657 2550 scope.go:117] "RemoveContainer" containerID="35944ea607085104207be82d284a44af4f91ab2aedfc82ec2a464914db6342c0" Jul 27 16:25:18.357210 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.357184 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d" Jul 27 16:25:18.362306 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.362273 2550 scope.go:117] "RemoveContainer" containerID="35944ea607085104207be82d284a44af4f91ab2aedfc82ec2a464914db6342c0" Jul 27 16:25:18.362609 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:25:18.362588 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35944ea607085104207be82d284a44af4f91ab2aedfc82ec2a464914db6342c0\": container with ID starting with 35944ea607085104207be82d284a44af4f91ab2aedfc82ec2a464914db6342c0 not found: ID does not exist" containerID="35944ea607085104207be82d284a44af4f91ab2aedfc82ec2a464914db6342c0" Jul 27 16:25:18.362700 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.362615 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35944ea607085104207be82d284a44af4f91ab2aedfc82ec2a464914db6342c0"} err="failed to get container status \"35944ea607085104207be82d284a44af4f91ab2aedfc82ec2a464914db6342c0\": rpc error: code = NotFound desc = could not find container \"35944ea607085104207be82d284a44af4f91ab2aedfc82ec2a464914db6342c0\": container with ID starting with 35944ea607085104207be82d284a44af4f91ab2aedfc82ec2a464914db6342c0 not found: ID does not exist" Jul 27 16:25:18.369670 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.369626 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mj5zs" podStartSLOduration=1.369609113 podStartE2EDuration="1.369609113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:25:18.369350762 +0000 UTC m=+497.041885262" watchObservedRunningTime="2026-07-27 16:25:18.369609113 +0000 UTC m=+497.042143614" Jul 27 16:25:18.391139 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.391063 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn" podStartSLOduration=1.391046426 podStartE2EDuration="1.391046426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:25:18.389153868 +0000 UTC m=+497.061688374" watchObservedRunningTime="2026-07-27 16:25:18.391046426 +0000 UTC m=+497.063580924" Jul 27 16:25:18.391536 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:25:18.391505 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-v267d\" is forbidden: User \"system:node:ip-10-0-135-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-70.ec2.internal' and this object" podUID="a5ce20e6-4303-4b07-a4e4-5844d49ab9d9" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d" Jul 27 16:25:18.446660 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.446624 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spp2q\" (UniqueName: \"kubernetes.io/projected/a5ce20e6-4303-4b07-a4e4-5844d49ab9d9-kube-api-access-spp2q\") pod \"a5ce20e6-4303-4b07-a4e4-5844d49ab9d9\" (UID: \"a5ce20e6-4303-4b07-a4e4-5844d49ab9d9\") " Jul 27 16:25:18.446854 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.446686 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5ce20e6-4303-4b07-a4e4-5844d49ab9d9-extensions-socket-volume\") pod \"a5ce20e6-4303-4b07-a4e4-5844d49ab9d9\" (UID: \"a5ce20e6-4303-4b07-a4e4-5844d49ab9d9\") " Jul 27 16:25:18.447028 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.446993 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ce20e6-4303-4b07-a4e4-5844d49ab9d9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a5ce20e6-4303-4b07-a4e4-5844d49ab9d9" (UID: "a5ce20e6-4303-4b07-a4e4-5844d49ab9d9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:25:18.447173 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.447107 2550 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5ce20e6-4303-4b07-a4e4-5844d49ab9d9-extensions-socket-volume\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:25:18.449173 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.449144 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ce20e6-4303-4b07-a4e4-5844d49ab9d9-kube-api-access-spp2q" (OuterVolumeSpecName: "kube-api-access-spp2q") pod "a5ce20e6-4303-4b07-a4e4-5844d49ab9d9" (UID: "a5ce20e6-4303-4b07-a4e4-5844d49ab9d9"). InnerVolumeSpecName "kube-api-access-spp2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:25:18.548314 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:18.548278 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spp2q\" (UniqueName: \"kubernetes.io/projected/a5ce20e6-4303-4b07-a4e4-5844d49ab9d9-kube-api-access-spp2q\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:25:19.354556 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:19.354502 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d" Jul 27 16:25:19.357249 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:25:19.357201 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-v267d\" is forbidden: User \"system:node:ip-10-0-135-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-70.ec2.internal' and this object" podUID="a5ce20e6-4303-4b07-a4e4-5844d49ab9d9" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d" Jul 27 16:25:19.363376 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:25:19.363344 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-v267d\" is forbidden: User \"system:node:ip-10-0-135-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-70.ec2.internal' and this object" podUID="a5ce20e6-4303-4b07-a4e4-5844d49ab9d9" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v267d" Jul 27 16:25:19.852885 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:19.852848 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5ce20e6-4303-4b07-a4e4-5844d49ab9d9" path="/var/lib/kubelet/pods/a5ce20e6-4303-4b07-a4e4-5844d49ab9d9/volumes" Jul 27 16:25:28.330079 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:28.330009 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d5c4dfbcd-8h8ft" podUID="59a1a815-2741-4e8e-a471-e9ebd3556a16" containerName="console" containerID="cri-o://826839c39a2a46f001611f41e16b2ec836007e1794d424218bba83d4bdc8a4d2" gracePeriod=15 Jul 27 16:25:29.356814 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:29.356784 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mj5zs" Jul 27 16:25:29.357284 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:29.357036 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn" Jul 27 16:25:29.653767 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:29.653734 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg"] Jul 27 16:25:29.874988 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:29.874953 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg"] Jul 27 16:25:29.875183 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:29.875036 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg" Jul 27 16:25:29.974928 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:29.974809 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89rzr\" (UniqueName: \"kubernetes.io/projected/4add1fc5-af54-42d9-afe0-999996159123-kube-api-access-89rzr\") pod \"kuadrant-operator-controller-manager-86878c769f-5vzpg\" (UID: \"4add1fc5-af54-42d9-afe0-999996159123\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg" Jul 27 16:25:29.975092 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:29.974986 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4add1fc5-af54-42d9-afe0-999996159123-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5vzpg\" (UID: \"4add1fc5-af54-42d9-afe0-999996159123\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg" Jul 27 16:25:30.076552 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:30.076511 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89rzr\" (UniqueName: \"kubernetes.io/projected/4add1fc5-af54-42d9-afe0-999996159123-kube-api-access-89rzr\") pod \"kuadrant-operator-controller-manager-86878c769f-5vzpg\" (UID: \"4add1fc5-af54-42d9-afe0-999996159123\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg" Jul 27 16:25:30.076740 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:30.076588 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4add1fc5-af54-42d9-afe0-999996159123-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5vzpg\" (UID: \"4add1fc5-af54-42d9-afe0-999996159123\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg" Jul 27 16:25:30.077050 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:30.077029 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4add1fc5-af54-42d9-afe0-999996159123-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5vzpg\" (UID: \"4add1fc5-af54-42d9-afe0-999996159123\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg" Jul 27 16:25:30.086689 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:30.086658 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89rzr\" (UniqueName: \"kubernetes.io/projected/4add1fc5-af54-42d9-afe0-999996159123-kube-api-access-89rzr\") pod \"kuadrant-operator-controller-manager-86878c769f-5vzpg\" (UID: \"4add1fc5-af54-42d9-afe0-999996159123\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg" Jul 27 16:25:30.188244 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:30.188199 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg" Jul 27 16:25:34.458658 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.458580 2550 generic.go:356] "Generic (PLEG): container finished" podID="59a1a815-2741-4e8e-a471-e9ebd3556a16" containerID="826839c39a2a46f001611f41e16b2ec836007e1794d424218bba83d4bdc8a4d2" exitCode=0 Jul 27 16:25:34.459073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.458656 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5c4dfbcd-8h8ft" event={"ID":"59a1a815-2741-4e8e-a471-e9ebd3556a16","Type":"ContainerDied","Data":"826839c39a2a46f001611f41e16b2ec836007e1794d424218bba83d4bdc8a4d2"} Jul 27 16:25:34.553129 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.553107 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg"] Jul 27 16:25:34.591501 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:25:34.591474 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4add1fc5_af54_42d9_afe0_999996159123.slice/crio-ddcb64e027610540e9a27ebe8d03c2c190fbaabaaff3ad117308e67279fd7aa0 WatchSource:0}: Error finding container ddcb64e027610540e9a27ebe8d03c2c190fbaabaaff3ad117308e67279fd7aa0: Status 404 returned error can't find the container with id ddcb64e027610540e9a27ebe8d03c2c190fbaabaaff3ad117308e67279fd7aa0 Jul 27 16:25:34.616729 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.616706 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:25:34.726232 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.726155 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-trusted-ca-bundle\") pod \"59a1a815-2741-4e8e-a471-e9ebd3556a16\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " Jul 27 16:25:34.726232 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.726228 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59a1a815-2741-4e8e-a471-e9ebd3556a16-console-oauth-config\") pod \"59a1a815-2741-4e8e-a471-e9ebd3556a16\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " Jul 27 16:25:34.726453 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.726335 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-service-ca\") pod \"59a1a815-2741-4e8e-a471-e9ebd3556a16\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " Jul 27 16:25:34.726453 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.726386 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59a1a815-2741-4e8e-a471-e9ebd3556a16-console-serving-cert\") pod \"59a1a815-2741-4e8e-a471-e9ebd3556a16\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " Jul 27 16:25:34.726453 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.726425 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-oauth-serving-cert\") pod \"59a1a815-2741-4e8e-a471-e9ebd3556a16\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " Jul 27 16:25:34.726610 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.726495 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fps5b\" (UniqueName: \"kubernetes.io/projected/59a1a815-2741-4e8e-a471-e9ebd3556a16-kube-api-access-fps5b\") pod \"59a1a815-2741-4e8e-a471-e9ebd3556a16\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " Jul 27 16:25:34.726610 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.726506 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "59a1a815-2741-4e8e-a471-e9ebd3556a16" (UID: "59a1a815-2741-4e8e-a471-e9ebd3556a16"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:25:34.726610 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.726532 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-console-config\") pod \"59a1a815-2741-4e8e-a471-e9ebd3556a16\" (UID: \"59a1a815-2741-4e8e-a471-e9ebd3556a16\") " Jul 27 16:25:34.726778 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.726646 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-service-ca" (OuterVolumeSpecName: "service-ca") pod "59a1a815-2741-4e8e-a471-e9ebd3556a16" (UID: "59a1a815-2741-4e8e-a471-e9ebd3556a16"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:25:34.726988 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.726859 2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-trusted-ca-bundle\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:25:34.726988 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.726882 2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-service-ca\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:25:34.726988 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.726947 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-console-config" (OuterVolumeSpecName: "console-config") pod "59a1a815-2741-4e8e-a471-e9ebd3556a16" (UID: "59a1a815-2741-4e8e-a471-e9ebd3556a16"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:25:34.727411 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.727383 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "59a1a815-2741-4e8e-a471-e9ebd3556a16" (UID: "59a1a815-2741-4e8e-a471-e9ebd3556a16"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:25:34.728582 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.728556 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a1a815-2741-4e8e-a471-e9ebd3556a16-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "59a1a815-2741-4e8e-a471-e9ebd3556a16" (UID: "59a1a815-2741-4e8e-a471-e9ebd3556a16"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 16:25:34.728679 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.728585 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a1a815-2741-4e8e-a471-e9ebd3556a16-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "59a1a815-2741-4e8e-a471-e9ebd3556a16" (UID: "59a1a815-2741-4e8e-a471-e9ebd3556a16"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 16:25:34.728679 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.728632 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a1a815-2741-4e8e-a471-e9ebd3556a16-kube-api-access-fps5b" (OuterVolumeSpecName: "kube-api-access-fps5b") pod "59a1a815-2741-4e8e-a471-e9ebd3556a16" (UID: "59a1a815-2741-4e8e-a471-e9ebd3556a16"). InnerVolumeSpecName "kube-api-access-fps5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:25:34.828087 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.828050 2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-console-config\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:25:34.828087 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.828079 2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59a1a815-2741-4e8e-a471-e9ebd3556a16-console-oauth-config\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:25:34.828087 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.828088 2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59a1a815-2741-4e8e-a471-e9ebd3556a16-console-serving-cert\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:25:34.828087 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.828096 2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59a1a815-2741-4e8e-a471-e9ebd3556a16-oauth-serving-cert\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:25:34.828346 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:34.828105 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fps5b\" (UniqueName: \"kubernetes.io/projected/59a1a815-2741-4e8e-a471-e9ebd3556a16-kube-api-access-fps5b\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:25:35.467700 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:35.467608 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg" event={"ID":"4add1fc5-af54-42d9-afe0-999996159123","Type":"ContainerStarted","Data":"ef7d70b6f3270b108848a700425f45de008d83592a50fd3070766676e315da19"} Jul 27 16:25:35.467700 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:35.467671 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg" event={"ID":"4add1fc5-af54-42d9-afe0-999996159123","Type":"ContainerStarted","Data":"ddcb64e027610540e9a27ebe8d03c2c190fbaabaaff3ad117308e67279fd7aa0"} Jul 27 16:25:35.468147 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:35.467805 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg" Jul 27 16:25:35.469261 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:35.469232 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dhx4s" event={"ID":"f67521c8-88be-43e7-9136-c7817310801f","Type":"ContainerStarted","Data":"6c5872df4a925475be528b139276f2a2b279e3aee66c3190b57820da2723ccda"} Jul 27 16:25:35.470516 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:35.470485 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5c4dfbcd-8h8ft" event={"ID":"59a1a815-2741-4e8e-a471-e9ebd3556a16","Type":"ContainerDied","Data":"031acdddfbf7a528346a2276e43249ed20b64cdc45296189062a6537c7a490f0"} Jul 27 16:25:35.470618 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:35.470532 2550 scope.go:117] "RemoveContainer" containerID="826839c39a2a46f001611f41e16b2ec836007e1794d424218bba83d4bdc8a4d2" Jul 27 16:25:35.470618 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:35.470556 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5c4dfbcd-8h8ft" Jul 27 16:25:35.493453 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:35.493410 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg" podStartSLOduration=6.493392985 podStartE2EDuration="6.493392985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:25:35.491395169 +0000 UTC m=+514.163929669" watchObservedRunningTime="2026-07-27 16:25:35.493392985 +0000 UTC m=+514.165927485" Jul 27 16:25:35.523668 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:35.523616 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dhx4s" podStartSLOduration=1.396379494 podStartE2EDuration="28.523599324s" totalImagesPullingTime="27.12721983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:25:35.521423159 +0000 UTC m=+514.193957658" watchObservedRunningTime="2026-07-27 16:25:35.523599324 +0000 UTC m=+514.196133824" Jul 27 16:25:35.544829 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:35.544792 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d5c4dfbcd-8h8ft"] Jul 27 16:25:35.551096 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:35.551064 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d5c4dfbcd-8h8ft"] Jul 27 16:25:35.850819 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:35.850786 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59a1a815-2741-4e8e-a471-e9ebd3556a16" path="/var/lib/kubelet/pods/59a1a815-2741-4e8e-a471-e9ebd3556a16/volumes" Jul 27 16:25:46.479948 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:46.479884 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg" Jul 27 16:25:46.560294 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:46.560255 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn"] Jul 27 16:25:46.560515 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:46.560491 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn" podUID="219d559f-a077-4984-92e3-bc87c11610b8" containerName="manager" containerID="cri-o://e85fe0aa615fa41ad1407401169970aecff2b8f675a46744bba6bc0fb7e5e82f" gracePeriod=10 Jul 27 16:25:46.819472 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:46.819442 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn" Jul 27 16:25:46.937569 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:46.937533 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tp9b\" (UniqueName: \"kubernetes.io/projected/219d559f-a077-4984-92e3-bc87c11610b8-kube-api-access-4tp9b\") pod \"219d559f-a077-4984-92e3-bc87c11610b8\" (UID: \"219d559f-a077-4984-92e3-bc87c11610b8\") " Jul 27 16:25:46.937771 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:46.937672 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/219d559f-a077-4984-92e3-bc87c11610b8-extensions-socket-volume\") pod \"219d559f-a077-4984-92e3-bc87c11610b8\" (UID: \"219d559f-a077-4984-92e3-bc87c11610b8\") " Jul 27 16:25:46.938118 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:46.938094 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219d559f-a077-4984-92e3-bc87c11610b8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "219d559f-a077-4984-92e3-bc87c11610b8" (UID: "219d559f-a077-4984-92e3-bc87c11610b8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:25:46.939850 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:46.939831 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219d559f-a077-4984-92e3-bc87c11610b8-kube-api-access-4tp9b" (OuterVolumeSpecName: "kube-api-access-4tp9b") pod "219d559f-a077-4984-92e3-bc87c11610b8" (UID: "219d559f-a077-4984-92e3-bc87c11610b8"). InnerVolumeSpecName "kube-api-access-4tp9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:25:47.039224 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:47.039186 2550 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/219d559f-a077-4984-92e3-bc87c11610b8-extensions-socket-volume\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:25:47.039224 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:47.039217 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tp9b\" (UniqueName: \"kubernetes.io/projected/219d559f-a077-4984-92e3-bc87c11610b8-kube-api-access-4tp9b\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:25:47.543045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:47.543010 2550 generic.go:356] "Generic (PLEG): container finished" podID="219d559f-a077-4984-92e3-bc87c11610b8" containerID="e85fe0aa615fa41ad1407401169970aecff2b8f675a46744bba6bc0fb7e5e82f" exitCode=0 Jul 27 16:25:47.543512 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:47.543064 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn" Jul 27 16:25:47.543512 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:47.543093 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn" event={"ID":"219d559f-a077-4984-92e3-bc87c11610b8","Type":"ContainerDied","Data":"e85fe0aa615fa41ad1407401169970aecff2b8f675a46744bba6bc0fb7e5e82f"} Jul 27 16:25:47.543512 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:47.543158 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn" event={"ID":"219d559f-a077-4984-92e3-bc87c11610b8","Type":"ContainerDied","Data":"a968e0762c319e554cfb75b624de4dc1c694b09f1a6bfdc7d96283d0bfde5115"} Jul 27 16:25:47.543512 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:47.543204 2550 scope.go:117] "RemoveContainer" containerID="e85fe0aa615fa41ad1407401169970aecff2b8f675a46744bba6bc0fb7e5e82f" Jul 27 16:25:47.553231 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:47.553215 2550 scope.go:117] "RemoveContainer" containerID="e85fe0aa615fa41ad1407401169970aecff2b8f675a46744bba6bc0fb7e5e82f" Jul 27 16:25:47.553491 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:25:47.553473 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e85fe0aa615fa41ad1407401169970aecff2b8f675a46744bba6bc0fb7e5e82f\": container with ID starting with e85fe0aa615fa41ad1407401169970aecff2b8f675a46744bba6bc0fb7e5e82f not found: ID does not exist" containerID="e85fe0aa615fa41ad1407401169970aecff2b8f675a46744bba6bc0fb7e5e82f" Jul 27 16:25:47.553563 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:47.553500 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e85fe0aa615fa41ad1407401169970aecff2b8f675a46744bba6bc0fb7e5e82f"} err="failed to get container status \"e85fe0aa615fa41ad1407401169970aecff2b8f675a46744bba6bc0fb7e5e82f\": rpc error: code = NotFound desc = could not find container \"e85fe0aa615fa41ad1407401169970aecff2b8f675a46744bba6bc0fb7e5e82f\": container with ID starting with e85fe0aa615fa41ad1407401169970aecff2b8f675a46744bba6bc0fb7e5e82f not found: ID does not exist" Jul 27 16:25:47.585225 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:47.585197 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn"] Jul 27 16:25:47.591347 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:47.591320 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jxlzn"] Jul 27 16:25:47.850108 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:25:47.850027 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="219d559f-a077-4984-92e3-bc87c11610b8" path="/var/lib/kubelet/pods/219d559f-a077-4984-92e3-bc87c11610b8/volumes" Jul 27 16:26:04.790452 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:04.790408 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7bfrw"] Jul 27 16:26:04.790843 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:04.790836 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59a1a815-2741-4e8e-a471-e9ebd3556a16" containerName="console" Jul 27 16:26:04.790890 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:04.790847 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="59a1a815-2741-4e8e-a471-e9ebd3556a16" containerName="console" Jul 27 16:26:04.790890 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:04.790874 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="219d559f-a077-4984-92e3-bc87c11610b8" containerName="manager" Jul 27 16:26:04.790890 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:04.790880 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="219d559f-a077-4984-92e3-bc87c11610b8" containerName="manager" Jul 27 16:26:04.791038 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:04.790994 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="59a1a815-2741-4e8e-a471-e9ebd3556a16" containerName="console" Jul 27 16:26:04.791038 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:04.791004 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="219d559f-a077-4984-92e3-bc87c11610b8" containerName="manager" Jul 27 16:26:04.904797 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:04.904711 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7bfrw"] Jul 27 16:26:04.905012 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:04.904801 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7bfrw" Jul 27 16:26:04.905012 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:04.904841 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7bfrw"] Jul 27 16:26:04.907973 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:04.907952 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 27 16:26:05.005896 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.005862 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/835f56e6-1bf6-4a8d-99b5-2e43517cc5b6-config-file\") pod \"limitador-limitador-64c94fd64f-7bfrw\" (UID: \"835f56e6-1bf6-4a8d-99b5-2e43517cc5b6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7bfrw" Jul 27 16:26:05.006082 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.005929 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhxqk\" (UniqueName: \"kubernetes.io/projected/835f56e6-1bf6-4a8d-99b5-2e43517cc5b6-kube-api-access-fhxqk\") pod \"limitador-limitador-64c94fd64f-7bfrw\" (UID: \"835f56e6-1bf6-4a8d-99b5-2e43517cc5b6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7bfrw" Jul 27 16:26:05.107447 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.107364 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/835f56e6-1bf6-4a8d-99b5-2e43517cc5b6-config-file\") pod \"limitador-limitador-64c94fd64f-7bfrw\" (UID: \"835f56e6-1bf6-4a8d-99b5-2e43517cc5b6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7bfrw" Jul 27 16:26:05.107447 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.107408 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhxqk\" (UniqueName: \"kubernetes.io/projected/835f56e6-1bf6-4a8d-99b5-2e43517cc5b6-kube-api-access-fhxqk\") pod \"limitador-limitador-64c94fd64f-7bfrw\" (UID: \"835f56e6-1bf6-4a8d-99b5-2e43517cc5b6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7bfrw" Jul 27 16:26:05.108245 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.108221 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/835f56e6-1bf6-4a8d-99b5-2e43517cc5b6-config-file\") pod \"limitador-limitador-64c94fd64f-7bfrw\" (UID: \"835f56e6-1bf6-4a8d-99b5-2e43517cc5b6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7bfrw" Jul 27 16:26:05.117243 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.117221 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhxqk\" (UniqueName: \"kubernetes.io/projected/835f56e6-1bf6-4a8d-99b5-2e43517cc5b6-kube-api-access-fhxqk\") pod \"limitador-limitador-64c94fd64f-7bfrw\" (UID: \"835f56e6-1bf6-4a8d-99b5-2e43517cc5b6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7bfrw" Jul 27 16:26:05.216235 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.216198 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7bfrw" Jul 27 16:26:05.387209 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.387184 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7bfrw"] Jul 27 16:26:05.389336 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:26:05.389305 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835f56e6_1bf6_4a8d_99b5_2e43517cc5b6.slice/crio-cef538823310a45ea67e60c9a9eb7614fa9dddcdf9ac00052d857d02891dc612 WatchSource:0}: Error finding container cef538823310a45ea67e60c9a9eb7614fa9dddcdf9ac00052d857d02891dc612: Status 404 returned error can't find the container with id cef538823310a45ea67e60c9a9eb7614fa9dddcdf9ac00052d857d02891dc612 Jul 27 16:26:05.651515 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.651430 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7bfrw" event={"ID":"835f56e6-1bf6-4a8d-99b5-2e43517cc5b6","Type":"ContainerStarted","Data":"cef538823310a45ea67e60c9a9eb7614fa9dddcdf9ac00052d857d02891dc612"} Jul 27 16:26:05.683172 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.683133 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-fsb6m"] Jul 27 16:26:05.691550 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.691008 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-fsb6m" Jul 27 16:26:05.693388 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.693351 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-fsb6m"] Jul 27 16:26:05.694491 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.694466 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gx6cq\"" Jul 27 16:26:05.813609 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.813574 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stzfm\" (UniqueName: \"kubernetes.io/projected/4fca415a-fef7-4f1c-981d-328758f337bb-kube-api-access-stzfm\") pod \"authorino-58cd955675-fsb6m\" (UID: \"4fca415a-fef7-4f1c-981d-328758f337bb\") " pod="kuadrant-system/authorino-58cd955675-fsb6m" Jul 27 16:26:05.825148 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.825115 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-9s589"] Jul 27 16:26:05.827926 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.827880 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9s589" Jul 27 16:26:05.837592 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.837569 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9s589"] Jul 27 16:26:05.915223 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.915124 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stzfm\" (UniqueName: \"kubernetes.io/projected/4fca415a-fef7-4f1c-981d-328758f337bb-kube-api-access-stzfm\") pod \"authorino-58cd955675-fsb6m\" (UID: \"4fca415a-fef7-4f1c-981d-328758f337bb\") " pod="kuadrant-system/authorino-58cd955675-fsb6m" Jul 27 16:26:05.915405 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.915256 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9hxp\" (UniqueName: \"kubernetes.io/projected/0150436c-bfe1-47e8-be8c-0695d1caf42c-kube-api-access-f9hxp\") pod \"authorino-75d556bd77-9s589\" (UID: \"0150436c-bfe1-47e8-be8c-0695d1caf42c\") " pod="kuadrant-system/authorino-75d556bd77-9s589" Jul 27 16:26:05.930858 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:05.930830 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stzfm\" (UniqueName: \"kubernetes.io/projected/4fca415a-fef7-4f1c-981d-328758f337bb-kube-api-access-stzfm\") pod \"authorino-58cd955675-fsb6m\" (UID: \"4fca415a-fef7-4f1c-981d-328758f337bb\") " pod="kuadrant-system/authorino-58cd955675-fsb6m" Jul 27 16:26:06.002740 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:06.002707 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-fsb6m" Jul 27 16:26:06.016721 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:06.016696 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9hxp\" (UniqueName: \"kubernetes.io/projected/0150436c-bfe1-47e8-be8c-0695d1caf42c-kube-api-access-f9hxp\") pod \"authorino-75d556bd77-9s589\" (UID: \"0150436c-bfe1-47e8-be8c-0695d1caf42c\") " pod="kuadrant-system/authorino-75d556bd77-9s589" Jul 27 16:26:06.030488 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:06.030458 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9hxp\" (UniqueName: \"kubernetes.io/projected/0150436c-bfe1-47e8-be8c-0695d1caf42c-kube-api-access-f9hxp\") pod \"authorino-75d556bd77-9s589\" (UID: \"0150436c-bfe1-47e8-be8c-0695d1caf42c\") " pod="kuadrant-system/authorino-75d556bd77-9s589" Jul 27 16:26:06.138033 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:06.137890 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9s589" Jul 27 16:26:06.192857 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:06.192779 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-fsb6m"] Jul 27 16:26:06.197171 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:26:06.197139 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fca415a_fef7_4f1c_981d_328758f337bb.slice/crio-5492f30034f10e059d2ad6202b8d83280bddc7e0c5426dc76243e353116f1070 WatchSource:0}: Error finding container 5492f30034f10e059d2ad6202b8d83280bddc7e0c5426dc76243e353116f1070: Status 404 returned error can't find the container with id 5492f30034f10e059d2ad6202b8d83280bddc7e0c5426dc76243e353116f1070 Jul 27 16:26:06.341813 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:06.341741 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9s589"] Jul 27 16:26:06.344377 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:26:06.344345 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0150436c_bfe1_47e8_be8c_0695d1caf42c.slice/crio-6a2e6ab219f39beb03541345775d1adfaf265337a9fdc36f67ce0bcd3bd0785d WatchSource:0}: Error finding container 6a2e6ab219f39beb03541345775d1adfaf265337a9fdc36f67ce0bcd3bd0785d: Status 404 returned error can't find the container with id 6a2e6ab219f39beb03541345775d1adfaf265337a9fdc36f67ce0bcd3bd0785d Jul 27 16:26:06.660012 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:06.659979 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-fsb6m" event={"ID":"4fca415a-fef7-4f1c-981d-328758f337bb","Type":"ContainerStarted","Data":"5492f30034f10e059d2ad6202b8d83280bddc7e0c5426dc76243e353116f1070"} Jul 27 16:26:06.661403 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:06.661376 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9s589" event={"ID":"0150436c-bfe1-47e8-be8c-0695d1caf42c","Type":"ContainerStarted","Data":"6a2e6ab219f39beb03541345775d1adfaf265337a9fdc36f67ce0bcd3bd0785d"} Jul 27 16:26:08.676128 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:08.676092 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7bfrw" event={"ID":"835f56e6-1bf6-4a8d-99b5-2e43517cc5b6","Type":"ContainerStarted","Data":"2601395d100e2e762b758cfdb9f1397e871a704a0cacd391755023ac0444f055"} Jul 27 16:26:08.676539 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:08.676267 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-7bfrw" Jul 27 16:26:08.695730 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:08.695676 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-7bfrw" podStartSLOduration=1.8360472749999999 podStartE2EDuration="4.695660453s" totalImagesPullingTime="2.859613178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:26:08.695638625 +0000 UTC m=+547.368173125" watchObservedRunningTime="2026-07-27 16:26:08.695660453 +0000 UTC m=+547.368194978" Jul 27 16:26:10.695789 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:10.695716 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9s589" event={"ID":"0150436c-bfe1-47e8-be8c-0695d1caf42c","Type":"ContainerStarted","Data":"909c5fbdcacfe3908e66848dac3be749590ce80f84343bd8192028e651929fa1"} Jul 27 16:26:10.697845 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:10.697814 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-fsb6m" event={"ID":"4fca415a-fef7-4f1c-981d-328758f337bb","Type":"ContainerStarted","Data":"65f1be96f836d56c4b567e8121c345e8e4c7af7bd2f15dc1e749285eff68e544"} Jul 27 16:26:10.719170 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:10.719105 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-9s589" podStartSLOduration=1.523376178 podStartE2EDuration="5.71908945s" totalImagesPullingTime="4.195713272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:26:10.716000478 +0000 UTC m=+549.388534980" watchObservedRunningTime="2026-07-27 16:26:10.71908945 +0000 UTC m=+549.391623948" Jul 27 16:26:10.736017 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:10.735953 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-fsb6m" podStartSLOduration=1.390918925 podStartE2EDuration="5.735894176s" totalImagesPullingTime="4.344975251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:26:10.733340839 +0000 UTC m=+549.405875339" watchObservedRunningTime="2026-07-27 16:26:10.735894176 +0000 UTC m=+549.408428676" Jul 27 16:26:10.761055 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:10.761028 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-fsb6m"] Jul 27 16:26:12.710032 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:12.709986 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-fsb6m" podUID="4fca415a-fef7-4f1c-981d-328758f337bb" containerName="authorino" containerID="cri-o://65f1be96f836d56c4b567e8121c345e8e4c7af7bd2f15dc1e749285eff68e544" gracePeriod=30 Jul 27 16:26:12.962120 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:12.962060 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-fsb6m" Jul 27 16:26:13.087973 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:13.087934 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stzfm\" (UniqueName: \"kubernetes.io/projected/4fca415a-fef7-4f1c-981d-328758f337bb-kube-api-access-stzfm\") pod \"4fca415a-fef7-4f1c-981d-328758f337bb\" (UID: \"4fca415a-fef7-4f1c-981d-328758f337bb\") " Jul 27 16:26:13.090292 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:13.090262 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fca415a-fef7-4f1c-981d-328758f337bb-kube-api-access-stzfm" (OuterVolumeSpecName: "kube-api-access-stzfm") pod "4fca415a-fef7-4f1c-981d-328758f337bb" (UID: "4fca415a-fef7-4f1c-981d-328758f337bb"). InnerVolumeSpecName "kube-api-access-stzfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:26:13.189563 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:13.189531 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stzfm\" (UniqueName: \"kubernetes.io/projected/4fca415a-fef7-4f1c-981d-328758f337bb-kube-api-access-stzfm\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:26:13.716674 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:13.716639 2550 generic.go:356] "Generic (PLEG): container finished" podID="4fca415a-fef7-4f1c-981d-328758f337bb" containerID="65f1be96f836d56c4b567e8121c345e8e4c7af7bd2f15dc1e749285eff68e544" exitCode=0 Jul 27 16:26:13.717128 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:13.716681 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-fsb6m" Jul 27 16:26:13.717128 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:13.716688 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-fsb6m" event={"ID":"4fca415a-fef7-4f1c-981d-328758f337bb","Type":"ContainerDied","Data":"65f1be96f836d56c4b567e8121c345e8e4c7af7bd2f15dc1e749285eff68e544"} Jul 27 16:26:13.717128 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:13.716729 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-fsb6m" event={"ID":"4fca415a-fef7-4f1c-981d-328758f337bb","Type":"ContainerDied","Data":"5492f30034f10e059d2ad6202b8d83280bddc7e0c5426dc76243e353116f1070"} Jul 27 16:26:13.717128 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:13.716746 2550 scope.go:117] "RemoveContainer" containerID="65f1be96f836d56c4b567e8121c345e8e4c7af7bd2f15dc1e749285eff68e544" Jul 27 16:26:13.726262 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:13.726244 2550 scope.go:117] "RemoveContainer" containerID="65f1be96f836d56c4b567e8121c345e8e4c7af7bd2f15dc1e749285eff68e544" Jul 27 16:26:13.726530 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:26:13.726512 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f1be96f836d56c4b567e8121c345e8e4c7af7bd2f15dc1e749285eff68e544\": container with ID starting with 65f1be96f836d56c4b567e8121c345e8e4c7af7bd2f15dc1e749285eff68e544 not found: ID does not exist" containerID="65f1be96f836d56c4b567e8121c345e8e4c7af7bd2f15dc1e749285eff68e544" Jul 27 16:26:13.726587 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:13.726540 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f1be96f836d56c4b567e8121c345e8e4c7af7bd2f15dc1e749285eff68e544"} err="failed to get container status \"65f1be96f836d56c4b567e8121c345e8e4c7af7bd2f15dc1e749285eff68e544\": rpc error: code = NotFound desc = could not find container \"65f1be96f836d56c4b567e8121c345e8e4c7af7bd2f15dc1e749285eff68e544\": container with ID starting with 65f1be96f836d56c4b567e8121c345e8e4c7af7bd2f15dc1e749285eff68e544 not found: ID does not exist" Jul 27 16:26:13.739395 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:13.739368 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-fsb6m"] Jul 27 16:26:13.741703 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:13.741683 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-fsb6m"] Jul 27 16:26:13.850385 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:13.850350 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fca415a-fef7-4f1c-981d-328758f337bb" path="/var/lib/kubelet/pods/4fca415a-fef7-4f1c-981d-328758f337bb/volumes" Jul 27 16:26:19.684757 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:19.684720 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-7bfrw" Jul 27 16:26:20.363071 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:20.363040 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7bfrw"] Jul 27 16:26:20.363270 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:20.363248 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-7bfrw" podUID="835f56e6-1bf6-4a8d-99b5-2e43517cc5b6" containerName="limitador" containerID="cri-o://2601395d100e2e762b758cfdb9f1397e871a704a0cacd391755023ac0444f055" gracePeriod=30 Jul 27 16:26:20.922421 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:20.922355 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7bfrw" Jul 27 16:26:21.057167 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:21.057136 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhxqk\" (UniqueName: \"kubernetes.io/projected/835f56e6-1bf6-4a8d-99b5-2e43517cc5b6-kube-api-access-fhxqk\") pod \"835f56e6-1bf6-4a8d-99b5-2e43517cc5b6\" (UID: \"835f56e6-1bf6-4a8d-99b5-2e43517cc5b6\") " Jul 27 16:26:21.057372 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:21.057217 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/835f56e6-1bf6-4a8d-99b5-2e43517cc5b6-config-file\") pod \"835f56e6-1bf6-4a8d-99b5-2e43517cc5b6\" (UID: \"835f56e6-1bf6-4a8d-99b5-2e43517cc5b6\") " Jul 27 16:26:21.057574 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:21.057543 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835f56e6-1bf6-4a8d-99b5-2e43517cc5b6-config-file" (OuterVolumeSpecName: "config-file") pod "835f56e6-1bf6-4a8d-99b5-2e43517cc5b6" (UID: "835f56e6-1bf6-4a8d-99b5-2e43517cc5b6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:26:21.059456 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:21.059435 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835f56e6-1bf6-4a8d-99b5-2e43517cc5b6-kube-api-access-fhxqk" (OuterVolumeSpecName: "kube-api-access-fhxqk") pod "835f56e6-1bf6-4a8d-99b5-2e43517cc5b6" (UID: "835f56e6-1bf6-4a8d-99b5-2e43517cc5b6"). InnerVolumeSpecName "kube-api-access-fhxqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:26:21.158338 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:21.158294 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhxqk\" (UniqueName: \"kubernetes.io/projected/835f56e6-1bf6-4a8d-99b5-2e43517cc5b6-kube-api-access-fhxqk\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:26:21.158338 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:21.158328 2550 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/835f56e6-1bf6-4a8d-99b5-2e43517cc5b6-config-file\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:26:21.773839 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:21.773798 2550 generic.go:356] "Generic (PLEG): container finished" podID="835f56e6-1bf6-4a8d-99b5-2e43517cc5b6" containerID="2601395d100e2e762b758cfdb9f1397e871a704a0cacd391755023ac0444f055" exitCode=0 Jul 27 16:26:21.774047 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:21.773860 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7bfrw" Jul 27 16:26:21.774047 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:21.773878 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7bfrw" event={"ID":"835f56e6-1bf6-4a8d-99b5-2e43517cc5b6","Type":"ContainerDied","Data":"2601395d100e2e762b758cfdb9f1397e871a704a0cacd391755023ac0444f055"} Jul 27 16:26:21.774047 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:21.773999 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7bfrw" event={"ID":"835f56e6-1bf6-4a8d-99b5-2e43517cc5b6","Type":"ContainerDied","Data":"cef538823310a45ea67e60c9a9eb7614fa9dddcdf9ac00052d857d02891dc612"} Jul 27 16:26:21.774948 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:21.774026 2550 scope.go:117] "RemoveContainer" containerID="2601395d100e2e762b758cfdb9f1397e871a704a0cacd391755023ac0444f055" Jul 27 16:26:21.784304 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:21.784287 2550 scope.go:117] "RemoveContainer" containerID="2601395d100e2e762b758cfdb9f1397e871a704a0cacd391755023ac0444f055" Jul 27 16:26:21.784563 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:26:21.784547 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2601395d100e2e762b758cfdb9f1397e871a704a0cacd391755023ac0444f055\": container with ID starting with 2601395d100e2e762b758cfdb9f1397e871a704a0cacd391755023ac0444f055 not found: ID does not exist" containerID="2601395d100e2e762b758cfdb9f1397e871a704a0cacd391755023ac0444f055" Jul 27 16:26:21.784606 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:21.784571 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2601395d100e2e762b758cfdb9f1397e871a704a0cacd391755023ac0444f055"} err="failed to get container status \"2601395d100e2e762b758cfdb9f1397e871a704a0cacd391755023ac0444f055\": rpc error: code = NotFound desc = could not find container \"2601395d100e2e762b758cfdb9f1397e871a704a0cacd391755023ac0444f055\": container with ID starting with 2601395d100e2e762b758cfdb9f1397e871a704a0cacd391755023ac0444f055 not found: ID does not exist" Jul 27 16:26:21.800036 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:21.800011 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7bfrw"] Jul 27 16:26:21.809433 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:21.809409 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7bfrw"] Jul 27 16:26:21.850752 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:21.850716 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="835f56e6-1bf6-4a8d-99b5-2e43517cc5b6" path="/var/lib/kubelet/pods/835f56e6-1bf6-4a8d-99b5-2e43517cc5b6/volumes" Jul 27 16:26:23.153192 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.153153 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9hkwj"] Jul 27 16:26:23.153829 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.153811 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="835f56e6-1bf6-4a8d-99b5-2e43517cc5b6" containerName="limitador" Jul 27 16:26:23.153873 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.153833 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="835f56e6-1bf6-4a8d-99b5-2e43517cc5b6" containerName="limitador" Jul 27 16:26:23.153873 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.153849 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fca415a-fef7-4f1c-981d-328758f337bb" containerName="authorino" Jul 27 16:26:23.153873 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.153857 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fca415a-fef7-4f1c-981d-328758f337bb" containerName="authorino" Jul 27 16:26:23.154006 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.153983 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="835f56e6-1bf6-4a8d-99b5-2e43517cc5b6" containerName="limitador" Jul 27 16:26:23.154006 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.154000 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fca415a-fef7-4f1c-981d-328758f337bb" containerName="authorino" Jul 27 16:26:23.156642 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.156622 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hkwj" Jul 27 16:26:23.159683 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.159656 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 27 16:26:23.159845 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.159704 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-x6pc5\"" Jul 27 16:26:23.159845 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.159731 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 16:26:23.160859 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.160827 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 27 16:26:23.166250 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.166229 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9hkwj"] Jul 27 16:26:23.278953 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.278898 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99mdk\" (UniqueName: \"kubernetes.io/projected/7b0eeb5c-d0f8-4573-b163-c2a54d1fbb54-kube-api-access-99mdk\") pod \"postgres-69fb4594cf-9hkwj\" (UID: \"7b0eeb5c-d0f8-4573-b163-c2a54d1fbb54\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hkwj" Jul 27 16:26:23.279167 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.279055 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7b0eeb5c-d0f8-4573-b163-c2a54d1fbb54-data\") pod \"postgres-69fb4594cf-9hkwj\" (UID: \"7b0eeb5c-d0f8-4573-b163-c2a54d1fbb54\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hkwj" Jul 27 16:26:23.379817 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.379776 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7b0eeb5c-d0f8-4573-b163-c2a54d1fbb54-data\") pod \"postgres-69fb4594cf-9hkwj\" (UID: \"7b0eeb5c-d0f8-4573-b163-c2a54d1fbb54\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hkwj" Jul 27 16:26:23.380023 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.379860 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99mdk\" (UniqueName: \"kubernetes.io/projected/7b0eeb5c-d0f8-4573-b163-c2a54d1fbb54-kube-api-access-99mdk\") pod \"postgres-69fb4594cf-9hkwj\" (UID: \"7b0eeb5c-d0f8-4573-b163-c2a54d1fbb54\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hkwj" Jul 27 16:26:23.380194 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.380176 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7b0eeb5c-d0f8-4573-b163-c2a54d1fbb54-data\") pod \"postgres-69fb4594cf-9hkwj\" (UID: \"7b0eeb5c-d0f8-4573-b163-c2a54d1fbb54\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hkwj" Jul 27 16:26:23.390119 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.390093 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99mdk\" (UniqueName: \"kubernetes.io/projected/7b0eeb5c-d0f8-4573-b163-c2a54d1fbb54-kube-api-access-99mdk\") pod \"postgres-69fb4594cf-9hkwj\" (UID: \"7b0eeb5c-d0f8-4573-b163-c2a54d1fbb54\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hkwj" Jul 27 16:26:23.468283 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.468185 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hkwj" Jul 27 16:26:23.614533 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.614501 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9hkwj"] Jul 27 16:26:23.614828 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:26:23.614805 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0eeb5c_d0f8_4573_b163_c2a54d1fbb54.slice/crio-4db66b617e428ffd8385788709168d9cd9849d715990e3508224a0b9a875b4c0 WatchSource:0}: Error finding container 4db66b617e428ffd8385788709168d9cd9849d715990e3508224a0b9a875b4c0: Status 404 returned error can't find the container with id 4db66b617e428ffd8385788709168d9cd9849d715990e3508224a0b9a875b4c0 Jul 27 16:26:23.787632 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:23.787595 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hkwj" event={"ID":"7b0eeb5c-d0f8-4573-b163-c2a54d1fbb54","Type":"ContainerStarted","Data":"4db66b617e428ffd8385788709168d9cd9849d715990e3508224a0b9a875b4c0"} Jul 27 16:26:28.824757 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:28.824719 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hkwj" event={"ID":"7b0eeb5c-d0f8-4573-b163-c2a54d1fbb54","Type":"ContainerStarted","Data":"9d9cd7c6fb3109906ef465131e4a53ec76751787f3ba77482ccf343e1748faa7"} Jul 27 16:26:28.825162 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:28.824943 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hkwj" Jul 27 16:26:28.848038 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:28.847977 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hkwj" podStartSLOduration=0.830553 podStartE2EDuration="5.847960268s" totalImagesPullingTime="5.017407268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:26:28.844118961 +0000 UTC m=+567.516653461" watchObservedRunningTime="2026-07-27 16:26:28.847960268 +0000 UTC m=+567.520494765" Jul 27 16:26:34.860164 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:34.860138 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9hkwj" Jul 27 16:26:35.404792 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:35.404762 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sjx8j"] Jul 27 16:26:35.408850 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:35.408828 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sjx8j" Jul 27 16:26:35.435223 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:35.435188 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sjx8j"] Jul 27 16:26:35.504889 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:35.504858 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf58l\" (UniqueName: \"kubernetes.io/projected/f0844429-8f64-4988-bab2-0f57e64cde06-kube-api-access-hf58l\") pod \"authorino-5d4b67c446-sjx8j\" (UID: \"f0844429-8f64-4988-bab2-0f57e64cde06\") " pod="kuadrant-system/authorino-5d4b67c446-sjx8j" Jul 27 16:26:35.606269 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:35.606232 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf58l\" (UniqueName: \"kubernetes.io/projected/f0844429-8f64-4988-bab2-0f57e64cde06-kube-api-access-hf58l\") pod \"authorino-5d4b67c446-sjx8j\" (UID: \"f0844429-8f64-4988-bab2-0f57e64cde06\") " pod="kuadrant-system/authorino-5d4b67c446-sjx8j" Jul 27 16:26:35.615613 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:35.615576 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf58l\" (UniqueName: \"kubernetes.io/projected/f0844429-8f64-4988-bab2-0f57e64cde06-kube-api-access-hf58l\") pod \"authorino-5d4b67c446-sjx8j\" (UID: \"f0844429-8f64-4988-bab2-0f57e64cde06\") " pod="kuadrant-system/authorino-5d4b67c446-sjx8j" Jul 27 16:26:35.629978 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:35.629942 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sjx8j"] Jul 27 16:26:35.630249 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:35.630236 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sjx8j" Jul 27 16:26:35.656392 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:35.656313 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-568c6f8b6-mdptt"] Jul 27 16:26:35.659764 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:35.659746 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-568c6f8b6-mdptt" Jul 27 16:26:35.670563 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:35.670525 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-568c6f8b6-mdptt"] Jul 27 16:26:35.707159 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:35.707112 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgzzs\" (UniqueName: \"kubernetes.io/projected/6b9ab125-2f09-4893-b6bc-412b44be4890-kube-api-access-tgzzs\") pod \"authorino-568c6f8b6-mdptt\" (UID: \"6b9ab125-2f09-4893-b6bc-412b44be4890\") " pod="kuadrant-system/authorino-568c6f8b6-mdptt" Jul 27 16:26:35.783379 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:35.783305 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sjx8j"] Jul 27 16:26:35.785764 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:26:35.785733 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0844429_8f64_4988_bab2_0f57e64cde06.slice/crio-8f27d09362144df0d3cf8ef2de5fc37dabc4db0e2135ba2d48839fe8af728143 WatchSource:0}: Error finding container 8f27d09362144df0d3cf8ef2de5fc37dabc4db0e2135ba2d48839fe8af728143: Status 404 returned error can't find the container with id 8f27d09362144df0d3cf8ef2de5fc37dabc4db0e2135ba2d48839fe8af728143 Jul 27 16:26:35.808473 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:35.808437 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgzzs\" (UniqueName: \"kubernetes.io/projected/6b9ab125-2f09-4893-b6bc-412b44be4890-kube-api-access-tgzzs\") pod \"authorino-568c6f8b6-mdptt\" (UID: \"6b9ab125-2f09-4893-b6bc-412b44be4890\") " pod="kuadrant-system/authorino-568c6f8b6-mdptt" Jul 27 16:26:35.826612 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:35.826583 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgzzs\" (UniqueName: \"kubernetes.io/projected/6b9ab125-2f09-4893-b6bc-412b44be4890-kube-api-access-tgzzs\") pod \"authorino-568c6f8b6-mdptt\" (UID: \"6b9ab125-2f09-4893-b6bc-412b44be4890\") " pod="kuadrant-system/authorino-568c6f8b6-mdptt" Jul 27 16:26:35.865942 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:35.865912 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sjx8j" event={"ID":"f0844429-8f64-4988-bab2-0f57e64cde06","Type":"ContainerStarted","Data":"8f27d09362144df0d3cf8ef2de5fc37dabc4db0e2135ba2d48839fe8af728143"} Jul 27 16:26:35.973646 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:35.973555 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-568c6f8b6-mdptt" Jul 27 16:26:36.127487 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:26:36.127450 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b9ab125_2f09_4893_b6bc_412b44be4890.slice/crio-e3977c4a6aa2a28c1bf04706ce80cbe64bc7b6ef6e988ee49e46942c90d0b226 WatchSource:0}: Error finding container e3977c4a6aa2a28c1bf04706ce80cbe64bc7b6ef6e988ee49e46942c90d0b226: Status 404 returned error can't find the container with id e3977c4a6aa2a28c1bf04706ce80cbe64bc7b6ef6e988ee49e46942c90d0b226 Jul 27 16:26:36.128219 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.128200 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-568c6f8b6-mdptt"] Jul 27 16:26:36.131716 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.131695 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-568c6f8b6-mdptt"] Jul 27 16:26:36.172110 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.172081 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bd57f78c8-q7d7t"] Jul 27 16:26:36.175477 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.175455 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd57f78c8-q7d7t" Jul 27 16:26:36.183679 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.183654 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 27 16:26:36.186442 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.186418 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bd57f78c8-q7d7t"] Jul 27 16:26:36.212281 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.212258 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6421413-ee66-4074-bdc0-776cf1e4faec-tls-cert\") pod \"authorino-bd57f78c8-q7d7t\" (UID: \"b6421413-ee66-4074-bdc0-776cf1e4faec\") " pod="kuadrant-system/authorino-bd57f78c8-q7d7t" Jul 27 16:26:36.212392 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.212343 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl476\" (UniqueName: \"kubernetes.io/projected/b6421413-ee66-4074-bdc0-776cf1e4faec-kube-api-access-sl476\") pod \"authorino-bd57f78c8-q7d7t\" (UID: \"b6421413-ee66-4074-bdc0-776cf1e4faec\") " pod="kuadrant-system/authorino-bd57f78c8-q7d7t" Jul 27 16:26:36.313616 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.313580 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6421413-ee66-4074-bdc0-776cf1e4faec-tls-cert\") pod \"authorino-bd57f78c8-q7d7t\" (UID: \"b6421413-ee66-4074-bdc0-776cf1e4faec\") " pod="kuadrant-system/authorino-bd57f78c8-q7d7t" Jul 27 16:26:36.313808 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.313655 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl476\" (UniqueName: \"kubernetes.io/projected/b6421413-ee66-4074-bdc0-776cf1e4faec-kube-api-access-sl476\") pod \"authorino-bd57f78c8-q7d7t\" (UID: \"b6421413-ee66-4074-bdc0-776cf1e4faec\") " pod="kuadrant-system/authorino-bd57f78c8-q7d7t" Jul 27 16:26:36.316399 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.316377 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6421413-ee66-4074-bdc0-776cf1e4faec-tls-cert\") pod \"authorino-bd57f78c8-q7d7t\" (UID: \"b6421413-ee66-4074-bdc0-776cf1e4faec\") " pod="kuadrant-system/authorino-bd57f78c8-q7d7t" Jul 27 16:26:36.324454 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.324430 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl476\" (UniqueName: \"kubernetes.io/projected/b6421413-ee66-4074-bdc0-776cf1e4faec-kube-api-access-sl476\") pod \"authorino-bd57f78c8-q7d7t\" (UID: \"b6421413-ee66-4074-bdc0-776cf1e4faec\") " pod="kuadrant-system/authorino-bd57f78c8-q7d7t" Jul 27 16:26:36.486146 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.486056 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd57f78c8-q7d7t" Jul 27 16:26:36.626750 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.626722 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bd57f78c8-q7d7t"] Jul 27 16:26:36.628425 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:26:36.628394 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6421413_ee66_4074_bdc0_776cf1e4faec.slice/crio-c5b38da875cc006d9bd385ac5f9ffd7ef8fe94068b8a41c65eb60de752ca9b6b WatchSource:0}: Error finding container c5b38da875cc006d9bd385ac5f9ffd7ef8fe94068b8a41c65eb60de752ca9b6b: Status 404 returned error can't find the container with id c5b38da875cc006d9bd385ac5f9ffd7ef8fe94068b8a41c65eb60de752ca9b6b Jul 27 16:26:36.874390 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.874345 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-568c6f8b6-mdptt" podUID="6b9ab125-2f09-4893-b6bc-412b44be4890" containerName="authorino" containerID="cri-o://dc0c5a72f4c40da57c307d96e3e1b5bb79594d670b89cb61c50ad613af9d7dc8" gracePeriod=30 Jul 27 16:26:36.874836 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.874396 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-568c6f8b6-mdptt" event={"ID":"6b9ab125-2f09-4893-b6bc-412b44be4890","Type":"ContainerStarted","Data":"dc0c5a72f4c40da57c307d96e3e1b5bb79594d670b89cb61c50ad613af9d7dc8"} Jul 27 16:26:36.874836 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.874468 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-568c6f8b6-mdptt" event={"ID":"6b9ab125-2f09-4893-b6bc-412b44be4890","Type":"ContainerStarted","Data":"e3977c4a6aa2a28c1bf04706ce80cbe64bc7b6ef6e988ee49e46942c90d0b226"} Jul 27 16:26:36.877996 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.877959 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd57f78c8-q7d7t" event={"ID":"b6421413-ee66-4074-bdc0-776cf1e4faec","Type":"ContainerStarted","Data":"c5b38da875cc006d9bd385ac5f9ffd7ef8fe94068b8a41c65eb60de752ca9b6b"} Jul 27 16:26:36.879820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.879793 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sjx8j" event={"ID":"f0844429-8f64-4988-bab2-0f57e64cde06","Type":"ContainerStarted","Data":"02002cb5be960c0704681cd41b3a9abee205bae59de7de75a801e20039fa368e"} Jul 27 16:26:36.879984 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.879922 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-sjx8j" podUID="f0844429-8f64-4988-bab2-0f57e64cde06" containerName="authorino" containerID="cri-o://02002cb5be960c0704681cd41b3a9abee205bae59de7de75a801e20039fa368e" gracePeriod=30 Jul 27 16:26:36.894139 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.894095 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-568c6f8b6-mdptt" podStartSLOduration=1.300853334 podStartE2EDuration="1.894079359s" totalImagesPullingTime="593.226025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:26:36.891335422 +0000 UTC m=+575.563869923" watchObservedRunningTime="2026-07-27 16:26:36.894079359 +0000 UTC m=+575.566613859" Jul 27 16:26:36.909421 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:36.909374 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-sjx8j" podStartSLOduration=1.497380299 podStartE2EDuration="1.909356817s" totalImagesPullingTime="411.976518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:26:36.908748383 +0000 UTC m=+575.581282881" watchObservedRunningTime="2026-07-27 16:26:36.909356817 +0000 UTC m=+575.581891319" Jul 27 16:26:37.131921 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.131877 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-568c6f8b6-mdptt" Jul 27 16:26:37.154687 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.154653 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sjx8j" Jul 27 16:26:37.221952 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.221920 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgzzs\" (UniqueName: \"kubernetes.io/projected/6b9ab125-2f09-4893-b6bc-412b44be4890-kube-api-access-tgzzs\") pod \"6b9ab125-2f09-4893-b6bc-412b44be4890\" (UID: \"6b9ab125-2f09-4893-b6bc-412b44be4890\") " Jul 27 16:26:37.222085 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.222015 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hf58l\" (UniqueName: \"kubernetes.io/projected/f0844429-8f64-4988-bab2-0f57e64cde06-kube-api-access-hf58l\") pod \"f0844429-8f64-4988-bab2-0f57e64cde06\" (UID: \"f0844429-8f64-4988-bab2-0f57e64cde06\") " Jul 27 16:26:37.224070 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.224041 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0844429-8f64-4988-bab2-0f57e64cde06-kube-api-access-hf58l" (OuterVolumeSpecName: "kube-api-access-hf58l") pod "f0844429-8f64-4988-bab2-0f57e64cde06" (UID: "f0844429-8f64-4988-bab2-0f57e64cde06"). InnerVolumeSpecName "kube-api-access-hf58l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:26:37.224169 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.224077 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9ab125-2f09-4893-b6bc-412b44be4890-kube-api-access-tgzzs" (OuterVolumeSpecName: "kube-api-access-tgzzs") pod "6b9ab125-2f09-4893-b6bc-412b44be4890" (UID: "6b9ab125-2f09-4893-b6bc-412b44be4890"). InnerVolumeSpecName "kube-api-access-tgzzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:26:37.322961 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.322930 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgzzs\" (UniqueName: \"kubernetes.io/projected/6b9ab125-2f09-4893-b6bc-412b44be4890-kube-api-access-tgzzs\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:26:37.322961 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.322957 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf58l\" (UniqueName: \"kubernetes.io/projected/f0844429-8f64-4988-bab2-0f57e64cde06-kube-api-access-hf58l\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:26:37.887885 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.887789 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd57f78c8-q7d7t" event={"ID":"b6421413-ee66-4074-bdc0-776cf1e4faec","Type":"ContainerStarted","Data":"972c0248cd6027870a3d431741f30ab23286444e74498b3c74db17d7e14b8082"} Jul 27 16:26:37.889168 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.889145 2550 generic.go:356] "Generic (PLEG): container finished" podID="f0844429-8f64-4988-bab2-0f57e64cde06" containerID="02002cb5be960c0704681cd41b3a9abee205bae59de7de75a801e20039fa368e" exitCode=0 Jul 27 16:26:37.889225 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.889175 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sjx8j" Jul 27 16:26:37.889278 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.889221 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sjx8j" event={"ID":"f0844429-8f64-4988-bab2-0f57e64cde06","Type":"ContainerDied","Data":"02002cb5be960c0704681cd41b3a9abee205bae59de7de75a801e20039fa368e"} Jul 27 16:26:37.889278 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.889253 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sjx8j" event={"ID":"f0844429-8f64-4988-bab2-0f57e64cde06","Type":"ContainerDied","Data":"8f27d09362144df0d3cf8ef2de5fc37dabc4db0e2135ba2d48839fe8af728143"} Jul 27 16:26:37.889370 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.889276 2550 scope.go:117] "RemoveContainer" containerID="02002cb5be960c0704681cd41b3a9abee205bae59de7de75a801e20039fa368e" Jul 27 16:26:37.890614 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.890594 2550 generic.go:356] "Generic (PLEG): container finished" podID="6b9ab125-2f09-4893-b6bc-412b44be4890" containerID="dc0c5a72f4c40da57c307d96e3e1b5bb79594d670b89cb61c50ad613af9d7dc8" exitCode=0 Jul 27 16:26:37.890708 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.890631 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-568c6f8b6-mdptt" Jul 27 16:26:37.890708 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.890640 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-568c6f8b6-mdptt" event={"ID":"6b9ab125-2f09-4893-b6bc-412b44be4890","Type":"ContainerDied","Data":"dc0c5a72f4c40da57c307d96e3e1b5bb79594d670b89cb61c50ad613af9d7dc8"} Jul 27 16:26:37.890708 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.890672 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-568c6f8b6-mdptt" event={"ID":"6b9ab125-2f09-4893-b6bc-412b44be4890","Type":"ContainerDied","Data":"e3977c4a6aa2a28c1bf04706ce80cbe64bc7b6ef6e988ee49e46942c90d0b226"} Jul 27 16:26:37.899010 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.898992 2550 scope.go:117] "RemoveContainer" containerID="02002cb5be960c0704681cd41b3a9abee205bae59de7de75a801e20039fa368e" Jul 27 16:26:37.899283 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:26:37.899265 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02002cb5be960c0704681cd41b3a9abee205bae59de7de75a801e20039fa368e\": container with ID starting with 02002cb5be960c0704681cd41b3a9abee205bae59de7de75a801e20039fa368e not found: ID does not exist" containerID="02002cb5be960c0704681cd41b3a9abee205bae59de7de75a801e20039fa368e" Jul 27 16:26:37.899371 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.899291 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02002cb5be960c0704681cd41b3a9abee205bae59de7de75a801e20039fa368e"} err="failed to get container status \"02002cb5be960c0704681cd41b3a9abee205bae59de7de75a801e20039fa368e\": rpc error: code = NotFound desc = could not find container \"02002cb5be960c0704681cd41b3a9abee205bae59de7de75a801e20039fa368e\": container with ID starting with 02002cb5be960c0704681cd41b3a9abee205bae59de7de75a801e20039fa368e not found: ID does not exist" Jul 27 16:26:37.899371 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.899306 2550 scope.go:117] "RemoveContainer" containerID="dc0c5a72f4c40da57c307d96e3e1b5bb79594d670b89cb61c50ad613af9d7dc8" Jul 27 16:26:37.907766 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.907736 2550 scope.go:117] "RemoveContainer" containerID="dc0c5a72f4c40da57c307d96e3e1b5bb79594d670b89cb61c50ad613af9d7dc8" Jul 27 16:26:37.908075 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:26:37.908053 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc0c5a72f4c40da57c307d96e3e1b5bb79594d670b89cb61c50ad613af9d7dc8\": container with ID starting with dc0c5a72f4c40da57c307d96e3e1b5bb79594d670b89cb61c50ad613af9d7dc8 not found: ID does not exist" containerID="dc0c5a72f4c40da57c307d96e3e1b5bb79594d670b89cb61c50ad613af9d7dc8" Jul 27 16:26:37.908134 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.908083 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0c5a72f4c40da57c307d96e3e1b5bb79594d670b89cb61c50ad613af9d7dc8"} err="failed to get container status \"dc0c5a72f4c40da57c307d96e3e1b5bb79594d670b89cb61c50ad613af9d7dc8\": rpc error: code = NotFound desc = could not find container \"dc0c5a72f4c40da57c307d96e3e1b5bb79594d670b89cb61c50ad613af9d7dc8\": container with ID starting with dc0c5a72f4c40da57c307d96e3e1b5bb79594d670b89cb61c50ad613af9d7dc8 not found: ID does not exist" Jul 27 16:26:37.909757 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.909717 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-bd57f78c8-q7d7t" podStartSLOduration=1.334630504 podStartE2EDuration="1.909706419s" totalImagesPullingTime="575.075915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:26:37.907294678 +0000 UTC m=+576.579829178" watchObservedRunningTime="2026-07-27 16:26:37.909706419 +0000 UTC m=+576.582240932" Jul 27 16:26:37.926430 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.926403 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sjx8j"] Jul 27 16:26:37.933023 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.932996 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sjx8j"] Jul 27 16:26:37.941078 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.941054 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9s589"] Jul 27 16:26:37.941251 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.941230 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-9s589" podUID="0150436c-bfe1-47e8-be8c-0695d1caf42c" containerName="authorino" containerID="cri-o://909c5fbdcacfe3908e66848dac3be749590ce80f84343bd8192028e651929fa1" gracePeriod=30 Jul 27 16:26:37.946377 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.946352 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-568c6f8b6-mdptt"] Jul 27 16:26:37.957027 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:37.956998 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-568c6f8b6-mdptt"] Jul 27 16:26:38.231877 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:38.231848 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9s589" Jul 27 16:26:38.332782 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:38.332754 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9hxp\" (UniqueName: \"kubernetes.io/projected/0150436c-bfe1-47e8-be8c-0695d1caf42c-kube-api-access-f9hxp\") pod \"0150436c-bfe1-47e8-be8c-0695d1caf42c\" (UID: \"0150436c-bfe1-47e8-be8c-0695d1caf42c\") " Jul 27 16:26:38.335139 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:38.335103 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0150436c-bfe1-47e8-be8c-0695d1caf42c-kube-api-access-f9hxp" (OuterVolumeSpecName: "kube-api-access-f9hxp") pod "0150436c-bfe1-47e8-be8c-0695d1caf42c" (UID: "0150436c-bfe1-47e8-be8c-0695d1caf42c"). InnerVolumeSpecName "kube-api-access-f9hxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:26:38.434292 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:38.434210 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9hxp\" (UniqueName: \"kubernetes.io/projected/0150436c-bfe1-47e8-be8c-0695d1caf42c-kube-api-access-f9hxp\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:26:38.899144 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:38.899107 2550 generic.go:356] "Generic (PLEG): container finished" podID="0150436c-bfe1-47e8-be8c-0695d1caf42c" containerID="909c5fbdcacfe3908e66848dac3be749590ce80f84343bd8192028e651929fa1" exitCode=0 Jul 27 16:26:38.899144 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:38.899149 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9s589" Jul 27 16:26:38.899648 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:38.899193 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9s589" event={"ID":"0150436c-bfe1-47e8-be8c-0695d1caf42c","Type":"ContainerDied","Data":"909c5fbdcacfe3908e66848dac3be749590ce80f84343bd8192028e651929fa1"} Jul 27 16:26:38.899648 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:38.899247 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9s589" event={"ID":"0150436c-bfe1-47e8-be8c-0695d1caf42c","Type":"ContainerDied","Data":"6a2e6ab219f39beb03541345775d1adfaf265337a9fdc36f67ce0bcd3bd0785d"} Jul 27 16:26:38.899648 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:38.899271 2550 scope.go:117] "RemoveContainer" containerID="909c5fbdcacfe3908e66848dac3be749590ce80f84343bd8192028e651929fa1" Jul 27 16:26:38.909171 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:38.909153 2550 scope.go:117] "RemoveContainer" containerID="909c5fbdcacfe3908e66848dac3be749590ce80f84343bd8192028e651929fa1" Jul 27 16:26:38.909442 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:26:38.909421 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"909c5fbdcacfe3908e66848dac3be749590ce80f84343bd8192028e651929fa1\": container with ID starting with 909c5fbdcacfe3908e66848dac3be749590ce80f84343bd8192028e651929fa1 not found: ID does not exist" containerID="909c5fbdcacfe3908e66848dac3be749590ce80f84343bd8192028e651929fa1" Jul 27 16:26:38.909513 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:38.909450 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"909c5fbdcacfe3908e66848dac3be749590ce80f84343bd8192028e651929fa1"} err="failed to get container status \"909c5fbdcacfe3908e66848dac3be749590ce80f84343bd8192028e651929fa1\": rpc error: code = NotFound desc = could not find container \"909c5fbdcacfe3908e66848dac3be749590ce80f84343bd8192028e651929fa1\": container with ID starting with 909c5fbdcacfe3908e66848dac3be749590ce80f84343bd8192028e651929fa1 not found: ID does not exist" Jul 27 16:26:38.921740 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:38.921711 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9s589"] Jul 27 16:26:38.925324 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:38.925288 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9s589"] Jul 27 16:26:39.851371 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:39.851343 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0150436c-bfe1-47e8-be8c-0695d1caf42c" path="/var/lib/kubelet/pods/0150436c-bfe1-47e8-be8c-0695d1caf42c/volumes" Jul 27 16:26:39.851655 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:39.851643 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b9ab125-2f09-4893-b6bc-412b44be4890" path="/var/lib/kubelet/pods/6b9ab125-2f09-4893-b6bc-412b44be4890/volumes" Jul 27 16:26:39.851943 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:26:39.851930 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0844429-8f64-4988-bab2-0f57e64cde06" path="/var/lib/kubelet/pods/f0844429-8f64-4988-bab2-0f57e64cde06/volumes" Jul 27 16:27:01.760233 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:01.760209 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:27:01.764107 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:01.764087 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:27:18.393792 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.393750 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-c7777c89c-g8vbl"] Jul 27 16:27:18.395328 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.394218 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0150436c-bfe1-47e8-be8c-0695d1caf42c" containerName="authorino" Jul 27 16:27:18.395328 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.394252 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0150436c-bfe1-47e8-be8c-0695d1caf42c" containerName="authorino" Jul 27 16:27:18.395328 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.394271 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0844429-8f64-4988-bab2-0f57e64cde06" containerName="authorino" Jul 27 16:27:18.395328 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.394276 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0844429-8f64-4988-bab2-0f57e64cde06" containerName="authorino" Jul 27 16:27:18.395328 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.394284 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b9ab125-2f09-4893-b6bc-412b44be4890" containerName="authorino" Jul 27 16:27:18.395328 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.394289 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b9ab125-2f09-4893-b6bc-412b44be4890" containerName="authorino" Jul 27 16:27:18.395328 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.394358 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0150436c-bfe1-47e8-be8c-0695d1caf42c" containerName="authorino" Jul 27 16:27:18.395328 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.394369 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b9ab125-2f09-4893-b6bc-412b44be4890" containerName="authorino" Jul 27 16:27:18.395328 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.394377 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0844429-8f64-4988-bab2-0f57e64cde06" containerName="authorino" Jul 27 16:27:18.396648 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.396633 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c7777c89c-g8vbl" Jul 27 16:27:18.400891 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.400873 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 27 16:27:18.401005 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.400989 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8zfn5\"" Jul 27 16:27:18.410123 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.410095 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c7777c89c-g8vbl"] Jul 27 16:27:18.417330 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.417307 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5f3323a-4152-4078-b02f-b8e7ead3e441-maas-api-tls\") pod \"maas-api-c7777c89c-g8vbl\" (UID: \"b5f3323a-4152-4078-b02f-b8e7ead3e441\") " pod="odh-ai-gateway-infra/maas-api-c7777c89c-g8vbl" Jul 27 16:27:18.417439 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.417335 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f3323a-4152-4078-b02f-b8e7ead3e441-openshift-service-ca\") pod \"maas-api-c7777c89c-g8vbl\" (UID: \"b5f3323a-4152-4078-b02f-b8e7ead3e441\") " pod="odh-ai-gateway-infra/maas-api-c7777c89c-g8vbl" Jul 27 16:27:18.417439 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.417357 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vj28\" (UniqueName: \"kubernetes.io/projected/b5f3323a-4152-4078-b02f-b8e7ead3e441-kube-api-access-6vj28\") pod \"maas-api-c7777c89c-g8vbl\" (UID: \"b5f3323a-4152-4078-b02f-b8e7ead3e441\") " pod="odh-ai-gateway-infra/maas-api-c7777c89c-g8vbl" Jul 27 16:27:18.518488 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.518457 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5f3323a-4152-4078-b02f-b8e7ead3e441-maas-api-tls\") pod \"maas-api-c7777c89c-g8vbl\" (UID: \"b5f3323a-4152-4078-b02f-b8e7ead3e441\") " pod="odh-ai-gateway-infra/maas-api-c7777c89c-g8vbl" Jul 27 16:27:18.518488 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.518490 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f3323a-4152-4078-b02f-b8e7ead3e441-openshift-service-ca\") pod \"maas-api-c7777c89c-g8vbl\" (UID: \"b5f3323a-4152-4078-b02f-b8e7ead3e441\") " pod="odh-ai-gateway-infra/maas-api-c7777c89c-g8vbl" Jul 27 16:27:18.518731 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.518514 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vj28\" (UniqueName: \"kubernetes.io/projected/b5f3323a-4152-4078-b02f-b8e7ead3e441-kube-api-access-6vj28\") pod \"maas-api-c7777c89c-g8vbl\" (UID: \"b5f3323a-4152-4078-b02f-b8e7ead3e441\") " pod="odh-ai-gateway-infra/maas-api-c7777c89c-g8vbl" Jul 27 16:27:18.519148 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.519129 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f3323a-4152-4078-b02f-b8e7ead3e441-openshift-service-ca\") pod \"maas-api-c7777c89c-g8vbl\" (UID: \"b5f3323a-4152-4078-b02f-b8e7ead3e441\") " pod="odh-ai-gateway-infra/maas-api-c7777c89c-g8vbl" Jul 27 16:27:18.521095 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.521073 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5f3323a-4152-4078-b02f-b8e7ead3e441-maas-api-tls\") pod \"maas-api-c7777c89c-g8vbl\" (UID: \"b5f3323a-4152-4078-b02f-b8e7ead3e441\") " pod="odh-ai-gateway-infra/maas-api-c7777c89c-g8vbl" Jul 27 16:27:18.526985 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.526967 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vj28\" (UniqueName: \"kubernetes.io/projected/b5f3323a-4152-4078-b02f-b8e7ead3e441-kube-api-access-6vj28\") pod \"maas-api-c7777c89c-g8vbl\" (UID: \"b5f3323a-4152-4078-b02f-b8e7ead3e441\") " pod="odh-ai-gateway-infra/maas-api-c7777c89c-g8vbl" Jul 27 16:27:18.707374 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.707291 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c7777c89c-g8vbl" Jul 27 16:27:18.759804 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.759550 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-kx6r6"] Jul 27 16:27:18.763390 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.763366 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kx6r6" Jul 27 16:27:18.767326 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.767302 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 27 16:27:18.767980 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.767608 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bqx2k\"" Jul 27 16:27:18.785069 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.785042 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-kx6r6"] Jul 27 16:27:18.821685 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.821651 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6e2c3b5-1303-4a7c-b111-2b0b661ff60c-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-kx6r6\" (UID: \"d6e2c3b5-1303-4a7c-b111-2b0b661ff60c\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kx6r6" Jul 27 16:27:18.821850 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.821694 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq2rf\" (UniqueName: \"kubernetes.io/projected/d6e2c3b5-1303-4a7c-b111-2b0b661ff60c-kube-api-access-kq2rf\") pod \"payload-pre-processing-687dbb6cf6-kx6r6\" (UID: \"d6e2c3b5-1303-4a7c-b111-2b0b661ff60c\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kx6r6" Jul 27 16:27:18.882844 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.882815 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c7777c89c-g8vbl"] Jul 27 16:27:18.884054 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:27:18.884025 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f3323a_4152_4078_b02f_b8e7ead3e441.slice/crio-0d3af1b50c4996fb9cf84f05d07a3e063eb508140a522fca03f9835c09576756 WatchSource:0}: Error finding container 0d3af1b50c4996fb9cf84f05d07a3e063eb508140a522fca03f9835c09576756: Status 404 returned error can't find the container with id 0d3af1b50c4996fb9cf84f05d07a3e063eb508140a522fca03f9835c09576756 Jul 27 16:27:18.885554 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.885536 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 16:27:18.922302 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.922264 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6e2c3b5-1303-4a7c-b111-2b0b661ff60c-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-kx6r6\" (UID: \"d6e2c3b5-1303-4a7c-b111-2b0b661ff60c\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kx6r6" Jul 27 16:27:18.922476 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.922304 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq2rf\" (UniqueName: \"kubernetes.io/projected/d6e2c3b5-1303-4a7c-b111-2b0b661ff60c-kube-api-access-kq2rf\") pod \"payload-pre-processing-687dbb6cf6-kx6r6\" (UID: \"d6e2c3b5-1303-4a7c-b111-2b0b661ff60c\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kx6r6" Jul 27 16:27:18.922957 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.922937 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6e2c3b5-1303-4a7c-b111-2b0b661ff60c-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-kx6r6\" (UID: \"d6e2c3b5-1303-4a7c-b111-2b0b661ff60c\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kx6r6" Jul 27 16:27:18.938808 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:18.938784 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq2rf\" (UniqueName: \"kubernetes.io/projected/d6e2c3b5-1303-4a7c-b111-2b0b661ff60c-kube-api-access-kq2rf\") pod \"payload-pre-processing-687dbb6cf6-kx6r6\" (UID: \"d6e2c3b5-1303-4a7c-b111-2b0b661ff60c\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kx6r6" Jul 27 16:27:19.081081 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:19.081049 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kx6r6" Jul 27 16:27:19.144085 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:19.144047 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c7777c89c-g8vbl" event={"ID":"b5f3323a-4152-4078-b02f-b8e7ead3e441","Type":"ContainerStarted","Data":"0d3af1b50c4996fb9cf84f05d07a3e063eb508140a522fca03f9835c09576756"} Jul 27 16:27:19.233211 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:19.233186 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-kx6r6"] Jul 27 16:27:19.234602 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:27:19.234574 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e2c3b5_1303_4a7c_b111_2b0b661ff60c.slice/crio-ab986c7f7dc668ee79f565695eb29a6ce1980c678582558d1813cc9593e4173d WatchSource:0}: Error finding container ab986c7f7dc668ee79f565695eb29a6ce1980c678582558d1813cc9593e4173d: Status 404 returned error can't find the container with id ab986c7f7dc668ee79f565695eb29a6ce1980c678582558d1813cc9593e4173d Jul 27 16:27:20.155383 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:20.155282 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kx6r6" event={"ID":"d6e2c3b5-1303-4a7c-b111-2b0b661ff60c","Type":"ContainerStarted","Data":"ab986c7f7dc668ee79f565695eb29a6ce1980c678582558d1813cc9593e4173d"} Jul 27 16:27:22.169192 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:22.169099 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c7777c89c-g8vbl" event={"ID":"b5f3323a-4152-4078-b02f-b8e7ead3e441","Type":"ContainerStarted","Data":"b7382a9627de53771d631ea4afe51269620bd3334141208ecc09e6215189925f"} Jul 27 16:27:22.169192 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:22.169161 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-c7777c89c-g8vbl" Jul 27 16:27:22.170765 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:22.170741 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kx6r6" event={"ID":"d6e2c3b5-1303-4a7c-b111-2b0b661ff60c","Type":"ContainerStarted","Data":"2b29af50aea6d02fafcf7131de6e40adce59db85b30366fc6ff08117616332a8"} Jul 27 16:27:22.170923 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:22.170880 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kx6r6" Jul 27 16:27:22.189029 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:22.188971 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-c7777c89c-g8vbl" podStartSLOduration=1.313542243 podStartE2EDuration="4.188955514s" totalImagesPullingTime="2.875413271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:27:22.187613073 +0000 UTC m=+620.860147571" watchObservedRunningTime="2026-07-27 16:27:22.188955514 +0000 UTC m=+620.861490013" Jul 27 16:27:22.208952 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:22.208871 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kx6r6" podStartSLOduration=1.683117438 podStartE2EDuration="4.208850331s" totalImagesPullingTime="2.525732893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:27:22.206207876 +0000 UTC m=+620.878742364" watchObservedRunningTime="2026-07-27 16:27:22.208850331 +0000 UTC m=+620.881384831" Jul 27 16:27:28.181666 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:28.181635 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-c7777c89c-g8vbl" Jul 27 16:27:33.178016 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:33.177987 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kx6r6" Jul 27 16:27:39.097680 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.097644 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65"] Jul 27 16:27:39.104323 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.104300 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.108088 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.108065 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 27 16:27:39.108221 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.108162 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 27 16:27:39.109369 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.109351 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tgshm\"" Jul 27 16:27:39.109442 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.109370 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 27 16:27:39.115619 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.115590 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65"] Jul 27 16:27:39.214583 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.214544 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e3481c-db1e-443d-af83-e7b0e46a1be0-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65\" (UID: \"a0e3481c-db1e-443d-af83-e7b0e46a1be0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.214759 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.214599 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e3481c-db1e-443d-af83-e7b0e46a1be0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65\" (UID: \"a0e3481c-db1e-443d-af83-e7b0e46a1be0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.214759 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.214677 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e3481c-db1e-443d-af83-e7b0e46a1be0-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65\" (UID: \"a0e3481c-db1e-443d-af83-e7b0e46a1be0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.214759 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.214704 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e3481c-db1e-443d-af83-e7b0e46a1be0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65\" (UID: \"a0e3481c-db1e-443d-af83-e7b0e46a1be0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.214872 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.214766 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e3481c-db1e-443d-af83-e7b0e46a1be0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65\" (UID: \"a0e3481c-db1e-443d-af83-e7b0e46a1be0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.214872 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.214806 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q85d\" (UniqueName: \"kubernetes.io/projected/a0e3481c-db1e-443d-af83-e7b0e46a1be0-kube-api-access-4q85d\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65\" (UID: \"a0e3481c-db1e-443d-af83-e7b0e46a1be0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.315707 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.315674 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e3481c-db1e-443d-af83-e7b0e46a1be0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65\" (UID: \"a0e3481c-db1e-443d-af83-e7b0e46a1be0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.315958 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.315736 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q85d\" (UniqueName: \"kubernetes.io/projected/a0e3481c-db1e-443d-af83-e7b0e46a1be0-kube-api-access-4q85d\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65\" (UID: \"a0e3481c-db1e-443d-af83-e7b0e46a1be0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.315958 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.315802 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e3481c-db1e-443d-af83-e7b0e46a1be0-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65\" (UID: \"a0e3481c-db1e-443d-af83-e7b0e46a1be0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.315958 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.315853 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e3481c-db1e-443d-af83-e7b0e46a1be0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65\" (UID: \"a0e3481c-db1e-443d-af83-e7b0e46a1be0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.315958 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.315896 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e3481c-db1e-443d-af83-e7b0e46a1be0-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65\" (UID: \"a0e3481c-db1e-443d-af83-e7b0e46a1be0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.315958 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.315952 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e3481c-db1e-443d-af83-e7b0e46a1be0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65\" (UID: \"a0e3481c-db1e-443d-af83-e7b0e46a1be0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.316256 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.316095 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e3481c-db1e-443d-af83-e7b0e46a1be0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65\" (UID: \"a0e3481c-db1e-443d-af83-e7b0e46a1be0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.316256 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.316162 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e3481c-db1e-443d-af83-e7b0e46a1be0-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65\" (UID: \"a0e3481c-db1e-443d-af83-e7b0e46a1be0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.316356 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.316284 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e3481c-db1e-443d-af83-e7b0e46a1be0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65\" (UID: \"a0e3481c-db1e-443d-af83-e7b0e46a1be0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.318423 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.318401 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e3481c-db1e-443d-af83-e7b0e46a1be0-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65\" (UID: \"a0e3481c-db1e-443d-af83-e7b0e46a1be0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.318580 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.318563 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e3481c-db1e-443d-af83-e7b0e46a1be0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65\" (UID: \"a0e3481c-db1e-443d-af83-e7b0e46a1be0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.325059 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.325034 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q85d\" (UniqueName: \"kubernetes.io/projected/a0e3481c-db1e-443d-af83-e7b0e46a1be0-kube-api-access-4q85d\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65\" (UID: \"a0e3481c-db1e-443d-af83-e7b0e46a1be0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.416098 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.416016 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:39.573073 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:39.573046 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65"] Jul 27 16:27:39.574565 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:27:39.574535 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e3481c_db1e_443d_af83_e7b0e46a1be0.slice/crio-eb4c02295254e439fb87d133dc46a3c8d0d6b64897e4fb69999a1474f79431e5 WatchSource:0}: Error finding container eb4c02295254e439fb87d133dc46a3c8d0d6b64897e4fb69999a1474f79431e5: Status 404 returned error can't find the container with id eb4c02295254e439fb87d133dc46a3c8d0d6b64897e4fb69999a1474f79431e5 Jul 27 16:27:40.287200 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:40.287161 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" event={"ID":"a0e3481c-db1e-443d-af83-e7b0e46a1be0","Type":"ContainerStarted","Data":"eb4c02295254e439fb87d133dc46a3c8d0d6b64897e4fb69999a1474f79431e5"} Jul 27 16:27:42.304135 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:42.304100 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" event={"ID":"a0e3481c-db1e-443d-af83-e7b0e46a1be0","Type":"ContainerStarted","Data":"bf1d07bf0bfe3c0559c79a5960c79cef6d074c204d0f2d49a544244545ab7f93"} Jul 27 16:27:42.327381 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:42.327320 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" podStartSLOduration=0.895002769 podStartE2EDuration="3.327300885s" totalImagesPullingTime="2.432298116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:27:42.324589166 +0000 UTC m=+640.997123666" watchObservedRunningTime="2026-07-27 16:27:42.327300885 +0000 UTC m=+640.999835385" Jul 27 16:27:46.856916 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:46.856876 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4"] Jul 27 16:27:46.861278 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:46.861259 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:46.865574 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:46.865546 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 27 16:27:46.873303 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:46.873282 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4"] Jul 27 16:27:46.990206 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:46.990164 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kfjb\" (UniqueName: \"kubernetes.io/projected/fb3eaca1-fe1c-47c8-b733-50dbcba620c2-kube-api-access-5kfjb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rmmz4\" (UID: \"fb3eaca1-fe1c-47c8-b733-50dbcba620c2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:46.990383 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:46.990283 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb3eaca1-fe1c-47c8-b733-50dbcba620c2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rmmz4\" (UID: \"fb3eaca1-fe1c-47c8-b733-50dbcba620c2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:46.990383 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:46.990316 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb3eaca1-fe1c-47c8-b733-50dbcba620c2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rmmz4\" (UID: \"fb3eaca1-fe1c-47c8-b733-50dbcba620c2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:46.990383 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:46.990335 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb3eaca1-fe1c-47c8-b733-50dbcba620c2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rmmz4\" (UID: \"fb3eaca1-fe1c-47c8-b733-50dbcba620c2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:46.990383 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:46.990374 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb3eaca1-fe1c-47c8-b733-50dbcba620c2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rmmz4\" (UID: \"fb3eaca1-fe1c-47c8-b733-50dbcba620c2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:46.990529 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:46.990425 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb3eaca1-fe1c-47c8-b733-50dbcba620c2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rmmz4\" (UID: \"fb3eaca1-fe1c-47c8-b733-50dbcba620c2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:47.091116 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:47.091082 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb3eaca1-fe1c-47c8-b733-50dbcba620c2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rmmz4\" (UID: \"fb3eaca1-fe1c-47c8-b733-50dbcba620c2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:47.091302 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:47.091139 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kfjb\" (UniqueName: \"kubernetes.io/projected/fb3eaca1-fe1c-47c8-b733-50dbcba620c2-kube-api-access-5kfjb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rmmz4\" (UID: \"fb3eaca1-fe1c-47c8-b733-50dbcba620c2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:47.091302 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:47.091218 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb3eaca1-fe1c-47c8-b733-50dbcba620c2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rmmz4\" (UID: \"fb3eaca1-fe1c-47c8-b733-50dbcba620c2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:47.091302 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:47.091239 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb3eaca1-fe1c-47c8-b733-50dbcba620c2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rmmz4\" (UID: \"fb3eaca1-fe1c-47c8-b733-50dbcba620c2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:47.091302 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:47.091255 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb3eaca1-fe1c-47c8-b733-50dbcba620c2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rmmz4\" (UID: \"fb3eaca1-fe1c-47c8-b733-50dbcba620c2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:47.091488 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:47.091307 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb3eaca1-fe1c-47c8-b733-50dbcba620c2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rmmz4\" (UID: \"fb3eaca1-fe1c-47c8-b733-50dbcba620c2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:47.091604 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:47.091580 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb3eaca1-fe1c-47c8-b733-50dbcba620c2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rmmz4\" (UID: \"fb3eaca1-fe1c-47c8-b733-50dbcba620c2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:47.091676 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:47.091600 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb3eaca1-fe1c-47c8-b733-50dbcba620c2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rmmz4\" (UID: \"fb3eaca1-fe1c-47c8-b733-50dbcba620c2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:47.091798 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:47.091778 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb3eaca1-fe1c-47c8-b733-50dbcba620c2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rmmz4\" (UID: \"fb3eaca1-fe1c-47c8-b733-50dbcba620c2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:47.093572 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:47.093552 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb3eaca1-fe1c-47c8-b733-50dbcba620c2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rmmz4\" (UID: \"fb3eaca1-fe1c-47c8-b733-50dbcba620c2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:47.093919 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:47.093882 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb3eaca1-fe1c-47c8-b733-50dbcba620c2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rmmz4\" (UID: \"fb3eaca1-fe1c-47c8-b733-50dbcba620c2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:47.104445 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:47.104416 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kfjb\" (UniqueName: \"kubernetes.io/projected/fb3eaca1-fe1c-47c8-b733-50dbcba620c2-kube-api-access-5kfjb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rmmz4\" (UID: \"fb3eaca1-fe1c-47c8-b733-50dbcba620c2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:47.172645 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:47.172555 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:47.326033 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:47.326002 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4"] Jul 27 16:27:47.327739 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:27:47.327709 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb3eaca1_fe1c_47c8_b733_50dbcba620c2.slice/crio-557762d639ba1a44b4c9fd53e7f7138863692b4ca1c786ffef80d381fa4911b0 WatchSource:0}: Error finding container 557762d639ba1a44b4c9fd53e7f7138863692b4ca1c786ffef80d381fa4911b0: Status 404 returned error can't find the container with id 557762d639ba1a44b4c9fd53e7f7138863692b4ca1c786ffef80d381fa4911b0 Jul 27 16:27:47.338623 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:47.338595 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" event={"ID":"fb3eaca1-fe1c-47c8-b733-50dbcba620c2","Type":"ContainerStarted","Data":"557762d639ba1a44b4c9fd53e7f7138863692b4ca1c786ffef80d381fa4911b0"} Jul 27 16:27:48.346175 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:48.346138 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" event={"ID":"fb3eaca1-fe1c-47c8-b733-50dbcba620c2","Type":"ContainerStarted","Data":"12d32beef1db75f6b6c2b23aa66324de60d11d7caf5fc7b4feea1b51bd939a8e"} Jul 27 16:27:48.367566 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:48.367517 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" podStartSLOduration=2.059413722 podStartE2EDuration="2.367499559s" totalImagesPullingTime="308.085837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:27:48.364619911 +0000 UTC m=+647.037154450" watchObservedRunningTime="2026-07-27 16:27:48.367499559 +0000 UTC m=+647.040034058" Jul 27 16:27:49.416681 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:49.416641 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:49.417100 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:49.416697 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:49.429844 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:49.429820 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:50.369019 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:50.368984 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65" Jul 27 16:27:57.173422 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:57.173378 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:57.173870 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:57.173441 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:57.186600 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:57.186573 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:27:57.411878 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:27:57.411851 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rmmz4" Jul 27 16:29:43.908117 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:43.908074 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-dc575bd5f-kwdkq"] Jul 27 16:29:43.915042 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:43.911942 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc575bd5f-kwdkq" Jul 27 16:29:43.924185 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:43.924159 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-dc575bd5f-kwdkq"] Jul 27 16:29:44.035645 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:44.035610 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg2wj\" (UniqueName: \"kubernetes.io/projected/cf79249e-3925-4569-ab1c-7944f8099652-kube-api-access-cg2wj\") pod \"authorino-dc575bd5f-kwdkq\" (UID: \"cf79249e-3925-4569-ab1c-7944f8099652\") " pod="kuadrant-system/authorino-dc575bd5f-kwdkq" Jul 27 16:29:44.035826 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:44.035692 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf79249e-3925-4569-ab1c-7944f8099652-tls-cert\") pod \"authorino-dc575bd5f-kwdkq\" (UID: \"cf79249e-3925-4569-ab1c-7944f8099652\") " pod="kuadrant-system/authorino-dc575bd5f-kwdkq" Jul 27 16:29:44.136841 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:44.136801 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf79249e-3925-4569-ab1c-7944f8099652-tls-cert\") pod \"authorino-dc575bd5f-kwdkq\" (UID: \"cf79249e-3925-4569-ab1c-7944f8099652\") " pod="kuadrant-system/authorino-dc575bd5f-kwdkq" Jul 27 16:29:44.137057 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:44.136922 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg2wj\" (UniqueName: \"kubernetes.io/projected/cf79249e-3925-4569-ab1c-7944f8099652-kube-api-access-cg2wj\") pod \"authorino-dc575bd5f-kwdkq\" (UID: \"cf79249e-3925-4569-ab1c-7944f8099652\") " pod="kuadrant-system/authorino-dc575bd5f-kwdkq" Jul 27 16:29:44.139588 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:44.139563 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf79249e-3925-4569-ab1c-7944f8099652-tls-cert\") pod \"authorino-dc575bd5f-kwdkq\" (UID: \"cf79249e-3925-4569-ab1c-7944f8099652\") " pod="kuadrant-system/authorino-dc575bd5f-kwdkq" Jul 27 16:29:44.151397 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:44.151364 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg2wj\" (UniqueName: \"kubernetes.io/projected/cf79249e-3925-4569-ab1c-7944f8099652-kube-api-access-cg2wj\") pod \"authorino-dc575bd5f-kwdkq\" (UID: \"cf79249e-3925-4569-ab1c-7944f8099652\") " pod="kuadrant-system/authorino-dc575bd5f-kwdkq" Jul 27 16:29:44.226820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:44.226713 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc575bd5f-kwdkq" Jul 27 16:29:44.390701 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:44.390674 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-dc575bd5f-kwdkq"] Jul 27 16:29:44.392819 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:29:44.392792 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf79249e_3925_4569_ab1c_7944f8099652.slice/crio-7f563446ffd5e55fea88722b82a1308d0647e8478b34ecc7dd3a511406d852b8 WatchSource:0}: Error finding container 7f563446ffd5e55fea88722b82a1308d0647e8478b34ecc7dd3a511406d852b8: Status 404 returned error can't find the container with id 7f563446ffd5e55fea88722b82a1308d0647e8478b34ecc7dd3a511406d852b8 Jul 27 16:29:45.078582 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:45.078543 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc575bd5f-kwdkq" event={"ID":"cf79249e-3925-4569-ab1c-7944f8099652","Type":"ContainerStarted","Data":"7f563446ffd5e55fea88722b82a1308d0647e8478b34ecc7dd3a511406d852b8"} Jul 27 16:29:46.087641 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:46.087605 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc575bd5f-kwdkq" event={"ID":"cf79249e-3925-4569-ab1c-7944f8099652","Type":"ContainerStarted","Data":"ed72449875697ceaaabe62e7a7974f6346d71d0cb9ea0d92c74ab3ce3659e658"} Jul 27 16:29:46.105852 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:46.105794 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-dc575bd5f-kwdkq" podStartSLOduration=1.577599363 podStartE2EDuration="3.105778333s" totalImagesPullingTime="1.52817897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:29:46.104882969 +0000 UTC m=+764.777417467" watchObservedRunningTime="2026-07-27 16:29:46.105778333 +0000 UTC m=+764.778312832" Jul 27 16:29:46.143803 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:46.143769 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bd57f78c8-q7d7t"] Jul 27 16:29:46.144022 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:46.143974 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-bd57f78c8-q7d7t" podUID="b6421413-ee66-4074-bdc0-776cf1e4faec" containerName="authorino" containerID="cri-o://972c0248cd6027870a3d431741f30ab23286444e74498b3c74db17d7e14b8082" gracePeriod=30 Jul 27 16:29:46.417323 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:46.417297 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd57f78c8-q7d7t" Jul 27 16:29:46.563804 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:46.563771 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6421413-ee66-4074-bdc0-776cf1e4faec-tls-cert\") pod \"b6421413-ee66-4074-bdc0-776cf1e4faec\" (UID: \"b6421413-ee66-4074-bdc0-776cf1e4faec\") " Jul 27 16:29:46.564004 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:46.563827 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl476\" (UniqueName: \"kubernetes.io/projected/b6421413-ee66-4074-bdc0-776cf1e4faec-kube-api-access-sl476\") pod \"b6421413-ee66-4074-bdc0-776cf1e4faec\" (UID: \"b6421413-ee66-4074-bdc0-776cf1e4faec\") " Jul 27 16:29:46.566214 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:46.566183 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6421413-ee66-4074-bdc0-776cf1e4faec-kube-api-access-sl476" (OuterVolumeSpecName: "kube-api-access-sl476") pod "b6421413-ee66-4074-bdc0-776cf1e4faec" (UID: "b6421413-ee66-4074-bdc0-776cf1e4faec"). InnerVolumeSpecName "kube-api-access-sl476". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:29:46.577580 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:46.577548 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6421413-ee66-4074-bdc0-776cf1e4faec-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "b6421413-ee66-4074-bdc0-776cf1e4faec" (UID: "b6421413-ee66-4074-bdc0-776cf1e4faec"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 16:29:46.665493 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:46.665461 2550 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6421413-ee66-4074-bdc0-776cf1e4faec-tls-cert\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:29:46.665493 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:46.665489 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl476\" (UniqueName: \"kubernetes.io/projected/b6421413-ee66-4074-bdc0-776cf1e4faec-kube-api-access-sl476\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:29:47.096654 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:47.096613 2550 generic.go:356] "Generic (PLEG): container finished" podID="b6421413-ee66-4074-bdc0-776cf1e4faec" containerID="972c0248cd6027870a3d431741f30ab23286444e74498b3c74db17d7e14b8082" exitCode=0 Jul 27 16:29:47.096654 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:47.096677 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd57f78c8-q7d7t" Jul 27 16:29:47.097375 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:47.097352 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd57f78c8-q7d7t" event={"ID":"b6421413-ee66-4074-bdc0-776cf1e4faec","Type":"ContainerDied","Data":"972c0248cd6027870a3d431741f30ab23286444e74498b3c74db17d7e14b8082"} Jul 27 16:29:47.097476 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:47.097397 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd57f78c8-q7d7t" event={"ID":"b6421413-ee66-4074-bdc0-776cf1e4faec","Type":"ContainerDied","Data":"c5b38da875cc006d9bd385ac5f9ffd7ef8fe94068b8a41c65eb60de752ca9b6b"} Jul 27 16:29:47.097476 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:47.097421 2550 scope.go:117] "RemoveContainer" containerID="972c0248cd6027870a3d431741f30ab23286444e74498b3c74db17d7e14b8082" Jul 27 16:29:47.109777 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:47.109756 2550 scope.go:117] "RemoveContainer" containerID="972c0248cd6027870a3d431741f30ab23286444e74498b3c74db17d7e14b8082" Jul 27 16:29:47.110109 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:29:47.110087 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"972c0248cd6027870a3d431741f30ab23286444e74498b3c74db17d7e14b8082\": container with ID starting with 972c0248cd6027870a3d431741f30ab23286444e74498b3c74db17d7e14b8082 not found: ID does not exist" containerID="972c0248cd6027870a3d431741f30ab23286444e74498b3c74db17d7e14b8082" Jul 27 16:29:47.110217 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:47.110117 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"972c0248cd6027870a3d431741f30ab23286444e74498b3c74db17d7e14b8082"} err="failed to get container status \"972c0248cd6027870a3d431741f30ab23286444e74498b3c74db17d7e14b8082\": rpc error: code = NotFound desc = could not find container \"972c0248cd6027870a3d431741f30ab23286444e74498b3c74db17d7e14b8082\": container with ID starting with 972c0248cd6027870a3d431741f30ab23286444e74498b3c74db17d7e14b8082 not found: ID does not exist" Jul 27 16:29:47.136764 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:47.136734 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bd57f78c8-q7d7t"] Jul 27 16:29:47.147123 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:47.147090 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-bd57f78c8-q7d7t"] Jul 27 16:29:47.851703 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:29:47.851661 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6421413-ee66-4074-bdc0-776cf1e4faec" path="/var/lib/kubelet/pods/b6421413-ee66-4074-bdc0-776cf1e4faec/volumes" Jul 27 16:32:01.811865 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:32:01.811837 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:32:01.811865 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:32:01.811838 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:36:38.541707 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:36:38.541616 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg"] Jul 27 16:36:38.542289 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:36:38.541860 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg" podUID="4add1fc5-af54-42d9-afe0-999996159123" containerName="manager" containerID="cri-o://ef7d70b6f3270b108848a700425f45de008d83592a50fd3070766676e315da19" gracePeriod=10 Jul 27 16:36:41.115770 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:36:41.115746 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg" Jul 27 16:36:41.203135 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:36:41.203046 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89rzr\" (UniqueName: \"kubernetes.io/projected/4add1fc5-af54-42d9-afe0-999996159123-kube-api-access-89rzr\") pod \"4add1fc5-af54-42d9-afe0-999996159123\" (UID: \"4add1fc5-af54-42d9-afe0-999996159123\") " Jul 27 16:36:41.203301 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:36:41.203225 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4add1fc5-af54-42d9-afe0-999996159123-extensions-socket-volume\") pod \"4add1fc5-af54-42d9-afe0-999996159123\" (UID: \"4add1fc5-af54-42d9-afe0-999996159123\") " Jul 27 16:36:41.203589 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:36:41.203568 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4add1fc5-af54-42d9-afe0-999996159123-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4add1fc5-af54-42d9-afe0-999996159123" (UID: "4add1fc5-af54-42d9-afe0-999996159123"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:36:41.205376 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:36:41.205347 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4add1fc5-af54-42d9-afe0-999996159123-kube-api-access-89rzr" (OuterVolumeSpecName: "kube-api-access-89rzr") pod "4add1fc5-af54-42d9-afe0-999996159123" (UID: "4add1fc5-af54-42d9-afe0-999996159123"). InnerVolumeSpecName "kube-api-access-89rzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:36:41.304700 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:36:41.304660 2550 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4add1fc5-af54-42d9-afe0-999996159123-extensions-socket-volume\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:36:41.304700 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:36:41.304697 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89rzr\" (UniqueName: \"kubernetes.io/projected/4add1fc5-af54-42d9-afe0-999996159123-kube-api-access-89rzr\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:36:41.649559 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:36:41.649523 2550 generic.go:356] "Generic (PLEG): container finished" podID="4add1fc5-af54-42d9-afe0-999996159123" containerID="ef7d70b6f3270b108848a700425f45de008d83592a50fd3070766676e315da19" exitCode=0 Jul 27 16:36:41.649735 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:36:41.649571 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg" Jul 27 16:36:41.649735 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:36:41.649612 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg" event={"ID":"4add1fc5-af54-42d9-afe0-999996159123","Type":"ContainerDied","Data":"ef7d70b6f3270b108848a700425f45de008d83592a50fd3070766676e315da19"} Jul 27 16:36:41.649735 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:36:41.649661 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg" event={"ID":"4add1fc5-af54-42d9-afe0-999996159123","Type":"ContainerDied","Data":"ddcb64e027610540e9a27ebe8d03c2c190fbaabaaff3ad117308e67279fd7aa0"} Jul 27 16:36:41.649735 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:36:41.649682 2550 scope.go:117] "RemoveContainer" containerID="ef7d70b6f3270b108848a700425f45de008d83592a50fd3070766676e315da19" Jul 27 16:36:41.660577 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:36:41.660563 2550 scope.go:117] "RemoveContainer" containerID="ef7d70b6f3270b108848a700425f45de008d83592a50fd3070766676e315da19" Jul 27 16:36:41.660837 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:36:41.660816 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef7d70b6f3270b108848a700425f45de008d83592a50fd3070766676e315da19\": container with ID starting with ef7d70b6f3270b108848a700425f45de008d83592a50fd3070766676e315da19 not found: ID does not exist" containerID="ef7d70b6f3270b108848a700425f45de008d83592a50fd3070766676e315da19" Jul 27 16:36:41.660971 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:36:41.660844 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef7d70b6f3270b108848a700425f45de008d83592a50fd3070766676e315da19"} err="failed to get container status \"ef7d70b6f3270b108848a700425f45de008d83592a50fd3070766676e315da19\": rpc error: code = NotFound desc = could not find container \"ef7d70b6f3270b108848a700425f45de008d83592a50fd3070766676e315da19\": container with ID starting with ef7d70b6f3270b108848a700425f45de008d83592a50fd3070766676e315da19 not found: ID does not exist" Jul 27 16:36:41.673741 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:36:41.673718 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg"] Jul 27 16:36:41.677645 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:36:41.677615 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5vzpg"] Jul 27 16:36:41.850414 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:36:41.850383 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4add1fc5-af54-42d9-afe0-999996159123" path="/var/lib/kubelet/pods/4add1fc5-af54-42d9-afe0-999996159123/volumes" Jul 27 16:37:01.852949 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:01.852917 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:37:01.854813 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:01.854791 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:37:39.713701 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:39.713661 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s62w"] Jul 27 16:37:39.714216 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:39.714150 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4add1fc5-af54-42d9-afe0-999996159123" containerName="manager" Jul 27 16:37:39.714216 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:39.714163 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4add1fc5-af54-42d9-afe0-999996159123" containerName="manager" Jul 27 16:37:39.714216 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:39.714191 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6421413-ee66-4074-bdc0-776cf1e4faec" containerName="authorino" Jul 27 16:37:39.714216 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:39.714198 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6421413-ee66-4074-bdc0-776cf1e4faec" containerName="authorino" Jul 27 16:37:39.714350 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:39.714277 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6421413-ee66-4074-bdc0-776cf1e4faec" containerName="authorino" Jul 27 16:37:39.714350 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:39.714285 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4add1fc5-af54-42d9-afe0-999996159123" containerName="manager" Jul 27 16:37:39.717660 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:39.717644 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s62w" Jul 27 16:37:39.720640 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:39.720617 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zjx4c\"" Jul 27 16:37:39.728529 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:39.728506 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s62w"] Jul 27 16:37:39.753517 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:39.753469 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86e77d7e-9cb7-468e-af10-06145a7a68c7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9s62w\" (UID: \"86e77d7e-9cb7-468e-af10-06145a7a68c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s62w" Jul 27 16:37:39.753694 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:39.753630 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md4pf\" (UniqueName: \"kubernetes.io/projected/86e77d7e-9cb7-468e-af10-06145a7a68c7-kube-api-access-md4pf\") pod \"kuadrant-operator-controller-manager-86878c769f-9s62w\" (UID: \"86e77d7e-9cb7-468e-af10-06145a7a68c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s62w" Jul 27 16:37:39.854419 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:39.854387 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md4pf\" (UniqueName: \"kubernetes.io/projected/86e77d7e-9cb7-468e-af10-06145a7a68c7-kube-api-access-md4pf\") pod \"kuadrant-operator-controller-manager-86878c769f-9s62w\" (UID: \"86e77d7e-9cb7-468e-af10-06145a7a68c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s62w" Jul 27 16:37:39.854573 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:39.854424 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86e77d7e-9cb7-468e-af10-06145a7a68c7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9s62w\" (UID: \"86e77d7e-9cb7-468e-af10-06145a7a68c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s62w" Jul 27 16:37:39.854758 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:39.854743 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86e77d7e-9cb7-468e-af10-06145a7a68c7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9s62w\" (UID: \"86e77d7e-9cb7-468e-af10-06145a7a68c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s62w" Jul 27 16:37:39.864278 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:39.864245 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md4pf\" (UniqueName: \"kubernetes.io/projected/86e77d7e-9cb7-468e-af10-06145a7a68c7-kube-api-access-md4pf\") pod \"kuadrant-operator-controller-manager-86878c769f-9s62w\" (UID: \"86e77d7e-9cb7-468e-af10-06145a7a68c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s62w" Jul 27 16:37:40.028523 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:40.028494 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s62w" Jul 27 16:37:40.175091 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:40.175064 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s62w"] Jul 27 16:37:40.178352 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:37:40.178322 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e77d7e_9cb7_468e_af10_06145a7a68c7.slice/crio-c88da6d241ef621939e9040c2c8c7fbca715b9ff0c8ef89e5f9b46fc3e1eb582 WatchSource:0}: Error finding container c88da6d241ef621939e9040c2c8c7fbca715b9ff0c8ef89e5f9b46fc3e1eb582: Status 404 returned error can't find the container with id c88da6d241ef621939e9040c2c8c7fbca715b9ff0c8ef89e5f9b46fc3e1eb582 Jul 27 16:37:41.015985 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:41.015947 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s62w" event={"ID":"86e77d7e-9cb7-468e-af10-06145a7a68c7","Type":"ContainerStarted","Data":"16bf239f3cc79f1079710ce4d2f11dd439335e68257fc8364a9a6d700076f84d"} Jul 27 16:37:41.016443 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:41.016007 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s62w" event={"ID":"86e77d7e-9cb7-468e-af10-06145a7a68c7","Type":"ContainerStarted","Data":"c88da6d241ef621939e9040c2c8c7fbca715b9ff0c8ef89e5f9b46fc3e1eb582"} Jul 27 16:37:41.016443 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:41.016082 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s62w" Jul 27 16:37:41.041852 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:41.041801 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s62w" podStartSLOduration=2.041787549 podStartE2EDuration="2.041787549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:37:41.038967534 +0000 UTC m=+1239.711502036" watchObservedRunningTime="2026-07-27 16:37:41.041787549 +0000 UTC m=+1239.714322047" Jul 27 16:37:52.024147 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:37:52.024110 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s62w" Jul 27 16:42:01.892941 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:42:01.892894 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:42:01.897119 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:42:01.897097 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:46:12.769822 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:12.769789 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a178d363-qd858"] Jul 27 16:46:12.773382 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:12.773366 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a178d363-qd858" Jul 27 16:46:12.776227 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:12.776208 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v2ffk\"" Jul 27 16:46:12.784353 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:12.784331 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a178d363-qd858"] Jul 27 16:46:12.813497 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:12.813469 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4474c5e0-0da3-4aa0-bb54-17dd16f174d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a178d363-qd858\" (UID: \"4474c5e0-0da3-4aa0-bb54-17dd16f174d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a178d363-qd858" Jul 27 16:46:12.914965 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:12.914928 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4474c5e0-0da3-4aa0-bb54-17dd16f174d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a178d363-qd858\" (UID: \"4474c5e0-0da3-4aa0-bb54-17dd16f174d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a178d363-qd858" Jul 27 16:46:12.915550 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:12.915531 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4474c5e0-0da3-4aa0-bb54-17dd16f174d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a178d363-qd858\" (UID: \"4474c5e0-0da3-4aa0-bb54-17dd16f174d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a178d363-qd858" Jul 27 16:46:13.084967 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:13.084857 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a178d363-qd858" Jul 27 16:46:13.232544 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:13.232518 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a178d363-qd858"] Jul 27 16:46:13.234467 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:46:13.234442 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4474c5e0_0da3_4aa0_bb54_17dd16f174d3.slice/crio-954b64ff8c9705fca53dbc228259dfae8290d70e4859612e9e410b5a2ba5dfa2 WatchSource:0}: Error finding container 954b64ff8c9705fca53dbc228259dfae8290d70e4859612e9e410b5a2ba5dfa2: Status 404 returned error can't find the container with id 954b64ff8c9705fca53dbc228259dfae8290d70e4859612e9e410b5a2ba5dfa2 Jul 27 16:46:13.236345 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:13.236329 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 16:46:14.155740 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:14.155703 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a178d363-qd858" event={"ID":"4474c5e0-0da3-4aa0-bb54-17dd16f174d3","Type":"ContainerStarted","Data":"954b64ff8c9705fca53dbc228259dfae8290d70e4859612e9e410b5a2ba5dfa2"} Jul 27 16:46:16.173269 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:16.173176 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a178d363-qd858" event={"ID":"4474c5e0-0da3-4aa0-bb54-17dd16f174d3","Type":"ContainerStarted","Data":"c733728c7ac77b71c6dfe53ef697e56f9665eeaff3fb56bc321c9493d28ba3b8"} Jul 27 16:46:16.195035 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:16.194990 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a178d363-qd858" podStartSLOduration=1.7834622530000002 podStartE2EDuration="4.194975236s" totalImagesPullingTime="2.411512983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:46:16.192306144 +0000 UTC m=+1754.864840644" watchObservedRunningTime="2026-07-27 16:46:16.194975236 +0000 UTC m=+1754.867509735" Jul 27 16:46:17.181224 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:17.181189 2550 generic.go:356] "Generic (PLEG): container finished" podID="4474c5e0-0da3-4aa0-bb54-17dd16f174d3" containerID="c733728c7ac77b71c6dfe53ef697e56f9665eeaff3fb56bc321c9493d28ba3b8" exitCode=0 Jul 27 16:46:17.181633 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:17.181264 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a178d363-qd858" event={"ID":"4474c5e0-0da3-4aa0-bb54-17dd16f174d3","Type":"ContainerDied","Data":"c733728c7ac77b71c6dfe53ef697e56f9665eeaff3fb56bc321c9493d28ba3b8"} Jul 27 16:46:18.336828 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:18.336806 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a178d363-qd858" Jul 27 16:46:18.472222 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:18.472135 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4474c5e0-0da3-4aa0-bb54-17dd16f174d3-maas-api-service-ca\") pod \"4474c5e0-0da3-4aa0-bb54-17dd16f174d3\" (UID: \"4474c5e0-0da3-4aa0-bb54-17dd16f174d3\") " Jul 27 16:46:18.472501 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:18.472477 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4474c5e0-0da3-4aa0-bb54-17dd16f174d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4474c5e0-0da3-4aa0-bb54-17dd16f174d3" (UID: "4474c5e0-0da3-4aa0-bb54-17dd16f174d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:46:18.573696 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:18.573652 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4474c5e0-0da3-4aa0-bb54-17dd16f174d3-maas-api-service-ca\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:46:19.194189 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:19.194144 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a178d363-qd858" event={"ID":"4474c5e0-0da3-4aa0-bb54-17dd16f174d3","Type":"ContainerDied","Data":"954b64ff8c9705fca53dbc228259dfae8290d70e4859612e9e410b5a2ba5dfa2"} Jul 27 16:46:19.194189 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:19.194176 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a178d363-qd858" Jul 27 16:46:19.194415 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:19.194201 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="954b64ff8c9705fca53dbc228259dfae8290d70e4859612e9e410b5a2ba5dfa2" Jul 27 16:46:27.841077 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:27.841035 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a178d363-qd858"] Jul 27 16:46:27.851105 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:27.851077 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a178d363-qd858"] Jul 27 16:46:29.851962 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:29.851929 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4474c5e0-0da3-4aa0-bb54-17dd16f174d3" path="/var/lib/kubelet/pods/4474c5e0-0da3-4aa0-bb54-17dd16f174d3/volumes" Jul 27 16:46:35.447668 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:35.447632 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p"] Jul 27 16:46:35.448104 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:35.448091 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4474c5e0-0da3-4aa0-bb54-17dd16f174d3" containerName="revoke-keys" Jul 27 16:46:35.448104 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:35.448106 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4474c5e0-0da3-4aa0-bb54-17dd16f174d3" containerName="revoke-keys" Jul 27 16:46:35.448212 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:35.448204 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4474c5e0-0da3-4aa0-bb54-17dd16f174d3" containerName="revoke-keys" Jul 27 16:46:35.453889 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:35.453871 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p" Jul 27 16:46:35.456811 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:35.456729 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a842e2ee\"" Jul 27 16:46:35.456962 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:35.456810 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a842e2ee-dockercfg-sfvtw\"" Jul 27 16:46:35.462251 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:35.462228 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p"] Jul 27 16:46:35.644301 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:35.644259 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c66f8575-87f6-4a8f-8812-ac7c8f83d52e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p\" (UID: \"c66f8575-87f6-4a8f-8812-ac7c8f83d52e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p" Jul 27 16:46:35.644515 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:35.644399 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nmtg\" (UniqueName: \"kubernetes.io/projected/c66f8575-87f6-4a8f-8812-ac7c8f83d52e-kube-api-access-4nmtg\") pod \"payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p\" (UID: \"c66f8575-87f6-4a8f-8812-ac7c8f83d52e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p" Jul 27 16:46:35.745605 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:35.745514 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c66f8575-87f6-4a8f-8812-ac7c8f83d52e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p\" (UID: \"c66f8575-87f6-4a8f-8812-ac7c8f83d52e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p" Jul 27 16:46:35.745768 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:35.745628 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nmtg\" (UniqueName: \"kubernetes.io/projected/c66f8575-87f6-4a8f-8812-ac7c8f83d52e-kube-api-access-4nmtg\") pod \"payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p\" (UID: \"c66f8575-87f6-4a8f-8812-ac7c8f83d52e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p" Jul 27 16:46:35.746276 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:35.746257 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c66f8575-87f6-4a8f-8812-ac7c8f83d52e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p\" (UID: \"c66f8575-87f6-4a8f-8812-ac7c8f83d52e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p" Jul 27 16:46:35.759538 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:35.759510 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nmtg\" (UniqueName: \"kubernetes.io/projected/c66f8575-87f6-4a8f-8812-ac7c8f83d52e-kube-api-access-4nmtg\") pod \"payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p\" (UID: \"c66f8575-87f6-4a8f-8812-ac7c8f83d52e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p" Jul 27 16:46:35.766255 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:35.766221 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p" Jul 27 16:46:35.930764 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:35.930740 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p"] Jul 27 16:46:35.933495 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:46:35.933445 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66f8575_87f6_4a8f_8812_ac7c8f83d52e.slice/crio-a86e7e4e1915d5f27991c473427caf0274e7d52f948d68d2680259efb011c7ab WatchSource:0}: Error finding container a86e7e4e1915d5f27991c473427caf0274e7d52f948d68d2680259efb011c7ab: Status 404 returned error can't find the container with id a86e7e4e1915d5f27991c473427caf0274e7d52f948d68d2680259efb011c7ab Jul 27 16:46:36.301868 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:36.301781 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p" event={"ID":"c66f8575-87f6-4a8f-8812-ac7c8f83d52e","Type":"ContainerStarted","Data":"9e714bbfdeef25efe96c5a9edcd6957f607e07e531884511b9be33bf57ff2543"} Jul 27 16:46:36.301868 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:36.301842 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p" event={"ID":"c66f8575-87f6-4a8f-8812-ac7c8f83d52e","Type":"ContainerStarted","Data":"a86e7e4e1915d5f27991c473427caf0274e7d52f948d68d2680259efb011c7ab"} Jul 27 16:46:36.302118 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:36.301915 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p" Jul 27 16:46:36.322304 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:36.322250 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p" podStartSLOduration=1.322234946 podStartE2EDuration="1.322234946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:46:36.319108804 +0000 UTC m=+1774.991643304" watchObservedRunningTime="2026-07-27 16:46:36.322234946 +0000 UTC m=+1774.994769444" Jul 27 16:46:47.311352 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:47.311317 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p" Jul 27 16:46:59.323354 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:59.323274 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4"] Jul 27 16:46:59.327176 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:59.327160 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4" Jul 27 16:46:59.330099 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:59.330081 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v2ffk\"" Jul 27 16:46:59.336525 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:59.336498 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4"] Jul 27 16:46:59.465951 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:59.465894 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1115422-051e-4495-a46f-9bd65cad89e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4\" (UID: \"f1115422-051e-4495-a46f-9bd65cad89e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4" Jul 27 16:46:59.566948 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:59.566897 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1115422-051e-4495-a46f-9bd65cad89e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4\" (UID: \"f1115422-051e-4495-a46f-9bd65cad89e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4" Jul 27 16:46:59.567529 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:59.567509 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1115422-051e-4495-a46f-9bd65cad89e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4\" (UID: \"f1115422-051e-4495-a46f-9bd65cad89e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4" Jul 27 16:46:59.639221 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:59.639141 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4" Jul 27 16:46:59.784702 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:46:59.784670 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4"] Jul 27 16:46:59.787656 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:46:59.787629 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1115422_051e_4495_a46f_9bd65cad89e4.slice/crio-e5b4c880928ae5cc71547be0359dedd943b95ad71b5ae67eb7e2fe15724fb652 WatchSource:0}: Error finding container e5b4c880928ae5cc71547be0359dedd943b95ad71b5ae67eb7e2fe15724fb652: Status 404 returned error can't find the container with id e5b4c880928ae5cc71547be0359dedd943b95ad71b5ae67eb7e2fe15724fb652 Jul 27 16:47:00.448402 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:00.448368 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4" event={"ID":"f1115422-051e-4495-a46f-9bd65cad89e4","Type":"ContainerStarted","Data":"9c1a8ca775cc7bfea0fa7efcc5d23707eed891728cfecb7004b6a28eb108382b"} Jul 27 16:47:00.448782 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:00.448450 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4" event={"ID":"f1115422-051e-4495-a46f-9bd65cad89e4","Type":"ContainerStarted","Data":"e5b4c880928ae5cc71547be0359dedd943b95ad71b5ae67eb7e2fe15724fb652"} Jul 27 16:47:00.469177 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:00.469135 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4" podStartSLOduration=1.469121745 podStartE2EDuration="1.469121745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:47:00.468171609 +0000 UTC m=+1799.140706104" watchObservedRunningTime="2026-07-27 16:47:00.469121745 +0000 UTC m=+1799.141656243" Jul 27 16:47:01.456540 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:01.456508 2550 generic.go:356] "Generic (PLEG): container finished" podID="f1115422-051e-4495-a46f-9bd65cad89e4" containerID="9c1a8ca775cc7bfea0fa7efcc5d23707eed891728cfecb7004b6a28eb108382b" exitCode=0 Jul 27 16:47:01.456964 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:01.456547 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4" event={"ID":"f1115422-051e-4495-a46f-9bd65cad89e4","Type":"ContainerDied","Data":"9c1a8ca775cc7bfea0fa7efcc5d23707eed891728cfecb7004b6a28eb108382b"} Jul 27 16:47:01.935919 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:01.935871 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:47:01.941867 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:01.941839 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:47:02.604386 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:02.604359 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4" Jul 27 16:47:02.699997 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:02.699955 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1115422-051e-4495-a46f-9bd65cad89e4-maas-api-service-ca\") pod \"f1115422-051e-4495-a46f-9bd65cad89e4\" (UID: \"f1115422-051e-4495-a46f-9bd65cad89e4\") " Jul 27 16:47:02.700378 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:02.700354 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1115422-051e-4495-a46f-9bd65cad89e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f1115422-051e-4495-a46f-9bd65cad89e4" (UID: "f1115422-051e-4495-a46f-9bd65cad89e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:47:02.801655 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:02.801622 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1115422-051e-4495-a46f-9bd65cad89e4-maas-api-service-ca\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:47:03.471054 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:03.471026 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4" Jul 27 16:47:03.471221 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:03.471024 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4" event={"ID":"f1115422-051e-4495-a46f-9bd65cad89e4","Type":"ContainerDied","Data":"e5b4c880928ae5cc71547be0359dedd943b95ad71b5ae67eb7e2fe15724fb652"} Jul 27 16:47:03.471221 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:03.471150 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5b4c880928ae5cc71547be0359dedd943b95ad71b5ae67eb7e2fe15724fb652" Jul 27 16:47:10.196936 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:10.196870 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p"] Jul 27 16:47:10.197433 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:10.197215 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p" podUID="c66f8575-87f6-4a8f-8812-ac7c8f83d52e" containerName="payload-pre-processing" containerID="cri-o://9e714bbfdeef25efe96c5a9edcd6957f607e07e531884511b9be33bf57ff2543" gracePeriod=30 Jul 27 16:47:10.464854 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:10.464784 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p" Jul 27 16:47:10.514711 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:10.514678 2550 generic.go:356] "Generic (PLEG): container finished" podID="c66f8575-87f6-4a8f-8812-ac7c8f83d52e" containerID="9e714bbfdeef25efe96c5a9edcd6957f607e07e531884511b9be33bf57ff2543" exitCode=0 Jul 27 16:47:10.514872 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:10.514727 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p" Jul 27 16:47:10.514872 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:10.514766 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p" event={"ID":"c66f8575-87f6-4a8f-8812-ac7c8f83d52e","Type":"ContainerDied","Data":"9e714bbfdeef25efe96c5a9edcd6957f607e07e531884511b9be33bf57ff2543"} Jul 27 16:47:10.514872 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:10.514824 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p" event={"ID":"c66f8575-87f6-4a8f-8812-ac7c8f83d52e","Type":"ContainerDied","Data":"a86e7e4e1915d5f27991c473427caf0274e7d52f948d68d2680259efb011c7ab"} Jul 27 16:47:10.514872 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:10.514848 2550 scope.go:117] "RemoveContainer" containerID="9e714bbfdeef25efe96c5a9edcd6957f607e07e531884511b9be33bf57ff2543" Jul 27 16:47:10.527212 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:10.527184 2550 scope.go:117] "RemoveContainer" containerID="9e714bbfdeef25efe96c5a9edcd6957f607e07e531884511b9be33bf57ff2543" Jul 27 16:47:10.527646 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:47:10.527627 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e714bbfdeef25efe96c5a9edcd6957f607e07e531884511b9be33bf57ff2543\": container with ID starting with 9e714bbfdeef25efe96c5a9edcd6957f607e07e531884511b9be33bf57ff2543 not found: ID does not exist" containerID="9e714bbfdeef25efe96c5a9edcd6957f607e07e531884511b9be33bf57ff2543" Jul 27 16:47:10.527706 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:10.527655 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e714bbfdeef25efe96c5a9edcd6957f607e07e531884511b9be33bf57ff2543"} err="failed to get container status \"9e714bbfdeef25efe96c5a9edcd6957f607e07e531884511b9be33bf57ff2543\": rpc error: code = NotFound desc = could not find container \"9e714bbfdeef25efe96c5a9edcd6957f607e07e531884511b9be33bf57ff2543\": container with ID starting with 9e714bbfdeef25efe96c5a9edcd6957f607e07e531884511b9be33bf57ff2543 not found: ID does not exist" Jul 27 16:47:10.577061 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:10.577028 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c66f8575-87f6-4a8f-8812-ac7c8f83d52e-plugins-config-volume\") pod \"c66f8575-87f6-4a8f-8812-ac7c8f83d52e\" (UID: \"c66f8575-87f6-4a8f-8812-ac7c8f83d52e\") " Jul 27 16:47:10.577191 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:10.577164 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nmtg\" (UniqueName: \"kubernetes.io/projected/c66f8575-87f6-4a8f-8812-ac7c8f83d52e-kube-api-access-4nmtg\") pod \"c66f8575-87f6-4a8f-8812-ac7c8f83d52e\" (UID: \"c66f8575-87f6-4a8f-8812-ac7c8f83d52e\") " Jul 27 16:47:10.577422 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:10.577402 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c66f8575-87f6-4a8f-8812-ac7c8f83d52e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c66f8575-87f6-4a8f-8812-ac7c8f83d52e" (UID: "c66f8575-87f6-4a8f-8812-ac7c8f83d52e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:47:10.579481 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:10.579454 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66f8575-87f6-4a8f-8812-ac7c8f83d52e-kube-api-access-4nmtg" (OuterVolumeSpecName: "kube-api-access-4nmtg") pod "c66f8575-87f6-4a8f-8812-ac7c8f83d52e" (UID: "c66f8575-87f6-4a8f-8812-ac7c8f83d52e"). InnerVolumeSpecName "kube-api-access-4nmtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:47:10.678213 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:10.678174 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nmtg\" (UniqueName: \"kubernetes.io/projected/c66f8575-87f6-4a8f-8812-ac7c8f83d52e-kube-api-access-4nmtg\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:47:10.678213 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:10.678206 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c66f8575-87f6-4a8f-8812-ac7c8f83d52e-plugins-config-volume\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:47:10.839155 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:10.839119 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p"] Jul 27 16:47:10.843618 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:10.843589 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a842e2ee-6794cddc88-vdm8p"] Jul 27 16:47:11.851449 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:11.851411 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c66f8575-87f6-4a8f-8812-ac7c8f83d52e" path="/var/lib/kubelet/pods/c66f8575-87f6-4a8f-8812-ac7c8f83d52e/volumes" Jul 27 16:47:14.409135 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:14.409097 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4"] Jul 27 16:47:14.412617 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:14.412594 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a842e2ee-qxqt4"] Jul 27 16:47:15.851500 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:15.851468 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1115422-051e-4495-a46f-9bd65cad89e4" path="/var/lib/kubelet/pods/f1115422-051e-4495-a46f-9bd65cad89e4/volumes" Jul 27 16:47:53.299016 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:53.298979 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5003e71d-dmpgn"] Jul 27 16:47:53.299491 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:53.299433 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1115422-051e-4495-a46f-9bd65cad89e4" containerName="revoke-keys" Jul 27 16:47:53.299491 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:53.299444 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1115422-051e-4495-a46f-9bd65cad89e4" containerName="revoke-keys" Jul 27 16:47:53.299491 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:53.299458 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66f8575-87f6-4a8f-8812-ac7c8f83d52e" containerName="payload-pre-processing" Jul 27 16:47:53.299491 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:53.299463 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66f8575-87f6-4a8f-8812-ac7c8f83d52e" containerName="payload-pre-processing" Jul 27 16:47:53.299621 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:53.299536 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c66f8575-87f6-4a8f-8812-ac7c8f83d52e" containerName="payload-pre-processing" Jul 27 16:47:53.299621 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:53.299547 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1115422-051e-4495-a46f-9bd65cad89e4" containerName="revoke-keys" Jul 27 16:47:53.302788 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:53.302770 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5003e71d-dmpgn" Jul 27 16:47:53.306340 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:53.306321 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v2ffk\"" Jul 27 16:47:53.312118 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:53.312084 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5003e71d-dmpgn"] Jul 27 16:47:53.389513 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:53.389482 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7741a7cc-c28a-4dc8-bb7e-2c6090aa135a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5003e71d-dmpgn\" (UID: \"7741a7cc-c28a-4dc8-bb7e-2c6090aa135a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5003e71d-dmpgn" Jul 27 16:47:53.491004 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:53.490967 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7741a7cc-c28a-4dc8-bb7e-2c6090aa135a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5003e71d-dmpgn\" (UID: \"7741a7cc-c28a-4dc8-bb7e-2c6090aa135a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5003e71d-dmpgn" Jul 27 16:47:53.491580 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:53.491559 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7741a7cc-c28a-4dc8-bb7e-2c6090aa135a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5003e71d-dmpgn\" (UID: \"7741a7cc-c28a-4dc8-bb7e-2c6090aa135a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5003e71d-dmpgn" Jul 27 16:47:53.614745 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:53.614661 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5003e71d-dmpgn" Jul 27 16:47:53.756232 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:53.756203 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5003e71d-dmpgn"] Jul 27 16:47:53.756854 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:47:53.756827 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7741a7cc_c28a_4dc8_bb7e_2c6090aa135a.slice/crio-cd6e83b8c8b89c677585de30787212e4926959fe6eb7bdc5f62ed1fc89adceb1 WatchSource:0}: Error finding container cd6e83b8c8b89c677585de30787212e4926959fe6eb7bdc5f62ed1fc89adceb1: Status 404 returned error can't find the container with id cd6e83b8c8b89c677585de30787212e4926959fe6eb7bdc5f62ed1fc89adceb1 Jul 27 16:47:53.783893 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:53.783869 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5003e71d-dmpgn" event={"ID":"7741a7cc-c28a-4dc8-bb7e-2c6090aa135a","Type":"ContainerStarted","Data":"cd6e83b8c8b89c677585de30787212e4926959fe6eb7bdc5f62ed1fc89adceb1"} Jul 27 16:47:54.791295 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:54.791258 2550 generic.go:356] "Generic (PLEG): container finished" podID="7741a7cc-c28a-4dc8-bb7e-2c6090aa135a" containerID="bce5e9da35822c5af4faa4463e7f8dc18aae54ded88c1ea31c6c071ec07e0640" exitCode=0 Jul 27 16:47:54.791663 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:54.791313 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5003e71d-dmpgn" event={"ID":"7741a7cc-c28a-4dc8-bb7e-2c6090aa135a","Type":"ContainerDied","Data":"bce5e9da35822c5af4faa4463e7f8dc18aae54ded88c1ea31c6c071ec07e0640"} Jul 27 16:47:55.941938 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:55.941914 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5003e71d-dmpgn" Jul 27 16:47:56.014632 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:56.014603 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7741a7cc-c28a-4dc8-bb7e-2c6090aa135a-maas-api-service-ca\") pod \"7741a7cc-c28a-4dc8-bb7e-2c6090aa135a\" (UID: \"7741a7cc-c28a-4dc8-bb7e-2c6090aa135a\") " Jul 27 16:47:56.014987 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:56.014961 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7741a7cc-c28a-4dc8-bb7e-2c6090aa135a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7741a7cc-c28a-4dc8-bb7e-2c6090aa135a" (UID: "7741a7cc-c28a-4dc8-bb7e-2c6090aa135a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:47:56.015136 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:56.015090 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7741a7cc-c28a-4dc8-bb7e-2c6090aa135a-maas-api-service-ca\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:47:56.805753 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:56.805726 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5003e71d-dmpgn" Jul 27 16:47:56.805936 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:56.805773 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5003e71d-dmpgn" event={"ID":"7741a7cc-c28a-4dc8-bb7e-2c6090aa135a","Type":"ContainerDied","Data":"cd6e83b8c8b89c677585de30787212e4926959fe6eb7bdc5f62ed1fc89adceb1"} Jul 27 16:47:56.805936 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:47:56.805848 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd6e83b8c8b89c677585de30787212e4926959fe6eb7bdc5f62ed1fc89adceb1" Jul 27 16:48:08.370063 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:08.370024 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5003e71d-dmpgn"] Jul 27 16:48:08.375681 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:08.375656 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5003e71d-dmpgn"] Jul 27 16:48:09.852482 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:09.852442 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7741a7cc-c28a-4dc8-bb7e-2c6090aa135a" path="/var/lib/kubelet/pods/7741a7cc-c28a-4dc8-bb7e-2c6090aa135a/volumes" Jul 27 16:48:14.218797 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.218761 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7"] Jul 27 16:48:14.219243 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.219230 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7741a7cc-c28a-4dc8-bb7e-2c6090aa135a" containerName="revoke-keys" Jul 27 16:48:14.219288 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.219245 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7741a7cc-c28a-4dc8-bb7e-2c6090aa135a" containerName="revoke-keys" Jul 27 16:48:14.219340 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.219332 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7741a7cc-c28a-4dc8-bb7e-2c6090aa135a" containerName="revoke-keys" Jul 27 16:48:14.223591 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.223577 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7" Jul 27 16:48:14.228067 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.228048 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-370eeced\"" Jul 27 16:48:14.246398 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.246369 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7"] Jul 27 16:48:14.282123 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.282092 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpvkd\" (UniqueName: \"kubernetes.io/projected/b93bd29d-fb81-447e-b61e-0909af95968e-kube-api-access-dpvkd\") pod \"payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7\" (UID: \"b93bd29d-fb81-447e-b61e-0909af95968e\") " pod="openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7" Jul 27 16:48:14.282285 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.282185 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b93bd29d-fb81-447e-b61e-0909af95968e-plugins-config-volume\") pod \"payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7\" (UID: \"b93bd29d-fb81-447e-b61e-0909af95968e\") " pod="openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7" Jul 27 16:48:14.315697 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.315660 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr"] Jul 27 16:48:14.319419 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.319402 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr" Jul 27 16:48:14.324320 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.324297 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-370eeced-dockercfg-d2v7l\"" Jul 27 16:48:14.341271 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.341246 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr"] Jul 27 16:48:14.382961 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.382923 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b93bd29d-fb81-447e-b61e-0909af95968e-plugins-config-volume\") pod \"payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7\" (UID: \"b93bd29d-fb81-447e-b61e-0909af95968e\") " pod="openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7" Jul 27 16:48:14.383125 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.383008 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292838de-77d4-40f1-a151-8d77bd64c927-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr\" (UID: \"292838de-77d4-40f1-a151-8d77bd64c927\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr" Jul 27 16:48:14.383125 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.383091 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpvkd\" (UniqueName: \"kubernetes.io/projected/b93bd29d-fb81-447e-b61e-0909af95968e-kube-api-access-dpvkd\") pod \"payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7\" (UID: \"b93bd29d-fb81-447e-b61e-0909af95968e\") " pod="openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7" Jul 27 16:48:14.383195 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.383148 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzr4k\" (UniqueName: \"kubernetes.io/projected/292838de-77d4-40f1-a151-8d77bd64c927-kube-api-access-bzr4k\") pod \"payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr\" (UID: \"292838de-77d4-40f1-a151-8d77bd64c927\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr" Jul 27 16:48:14.383654 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.383637 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b93bd29d-fb81-447e-b61e-0909af95968e-plugins-config-volume\") pod \"payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7\" (UID: \"b93bd29d-fb81-447e-b61e-0909af95968e\") " pod="openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7" Jul 27 16:48:14.394637 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.394463 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpvkd\" (UniqueName: \"kubernetes.io/projected/b93bd29d-fb81-447e-b61e-0909af95968e-kube-api-access-dpvkd\") pod \"payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7\" (UID: \"b93bd29d-fb81-447e-b61e-0909af95968e\") " pod="openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7" Jul 27 16:48:14.484543 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.484459 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzr4k\" (UniqueName: \"kubernetes.io/projected/292838de-77d4-40f1-a151-8d77bd64c927-kube-api-access-bzr4k\") pod \"payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr\" (UID: \"292838de-77d4-40f1-a151-8d77bd64c927\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr" Jul 27 16:48:14.484543 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.484537 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292838de-77d4-40f1-a151-8d77bd64c927-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr\" (UID: \"292838de-77d4-40f1-a151-8d77bd64c927\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr" Jul 27 16:48:14.485236 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.485219 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292838de-77d4-40f1-a151-8d77bd64c927-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr\" (UID: \"292838de-77d4-40f1-a151-8d77bd64c927\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr" Jul 27 16:48:14.498564 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.498530 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzr4k\" (UniqueName: \"kubernetes.io/projected/292838de-77d4-40f1-a151-8d77bd64c927-kube-api-access-bzr4k\") pod \"payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr\" (UID: \"292838de-77d4-40f1-a151-8d77bd64c927\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr" Jul 27 16:48:14.534699 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.534664 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7" Jul 27 16:48:14.630440 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.630239 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr" Jul 27 16:48:14.686334 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.686305 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7"] Jul 27 16:48:14.689548 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:48:14.689395 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93bd29d_fb81_447e_b61e_0909af95968e.slice/crio-dfd560293073e9432f324c218c33f6847ddb4e5e335050959354dc85d6357551 WatchSource:0}: Error finding container dfd560293073e9432f324c218c33f6847ddb4e5e335050959354dc85d6357551: Status 404 returned error can't find the container with id dfd560293073e9432f324c218c33f6847ddb4e5e335050959354dc85d6357551 Jul 27 16:48:14.813934 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.813877 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr"] Jul 27 16:48:14.815723 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:48:14.815700 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292838de_77d4_40f1_a151_8d77bd64c927.slice/crio-cd5b00085b37018a58253cec1a8748fd4ee834c6839b6723593f4652cbf01219 WatchSource:0}: Error finding container cd5b00085b37018a58253cec1a8748fd4ee834c6839b6723593f4652cbf01219: Status 404 returned error can't find the container with id cd5b00085b37018a58253cec1a8748fd4ee834c6839b6723593f4652cbf01219 Jul 27 16:48:14.923003 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.922964 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr" event={"ID":"292838de-77d4-40f1-a151-8d77bd64c927","Type":"ContainerStarted","Data":"06ed842bc580153843fb41849c59740f5dcaea8507ba0cdd34f70cdd3cdc4bc5"} Jul 27 16:48:14.923192 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.923033 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr" Jul 27 16:48:14.923192 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.923059 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr" event={"ID":"292838de-77d4-40f1-a151-8d77bd64c927","Type":"ContainerStarted","Data":"cd5b00085b37018a58253cec1a8748fd4ee834c6839b6723593f4652cbf01219"} Jul 27 16:48:14.924606 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.924577 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7" event={"ID":"b93bd29d-fb81-447e-b61e-0909af95968e","Type":"ContainerStarted","Data":"321f56b0cc801ca75e9eadc45c12a416088556d78c6a8cec9e5e6d98536fee37"} Jul 27 16:48:14.924736 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.924629 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7" event={"ID":"b93bd29d-fb81-447e-b61e-0909af95968e","Type":"ContainerStarted","Data":"dfd560293073e9432f324c218c33f6847ddb4e5e335050959354dc85d6357551"} Jul 27 16:48:14.924736 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.924651 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7" Jul 27 16:48:14.946762 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.946706 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr" podStartSLOduration=0.946688893 podStartE2EDuration="946.688893ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:48:14.944994948 +0000 UTC m=+1873.617529448" watchObservedRunningTime="2026-07-27 16:48:14.946688893 +0000 UTC m=+1873.619223391" Jul 27 16:48:14.968754 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:14.968699 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7" podStartSLOduration=0.968684776 podStartE2EDuration="968.684776ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:48:14.966071828 +0000 UTC m=+1873.638606327" watchObservedRunningTime="2026-07-27 16:48:14.968684776 +0000 UTC m=+1873.641219274" Jul 27 16:48:25.934036 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:25.933954 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7" Jul 27 16:48:25.934036 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:25.934035 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr" Jul 27 16:48:58.085932 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:58.078173 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7"] Jul 27 16:48:58.085932 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:58.078503 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7" podUID="b93bd29d-fb81-447e-b61e-0909af95968e" containerName="payload-processing" containerID="cri-o://321f56b0cc801ca75e9eadc45c12a416088556d78c6a8cec9e5e6d98536fee37" gracePeriod=30 Jul 27 16:48:58.090424 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:58.090394 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr"] Jul 27 16:48:58.090863 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:58.090814 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr" podUID="292838de-77d4-40f1-a151-8d77bd64c927" containerName="payload-pre-processing" containerID="cri-o://06ed842bc580153843fb41849c59740f5dcaea8507ba0cdd34f70cdd3cdc4bc5" gracePeriod=30 Jul 27 16:48:58.376037 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:58.376006 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7" Jul 27 16:48:58.379260 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:58.379242 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr" Jul 27 16:48:58.423727 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:58.423698 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292838de-77d4-40f1-a151-8d77bd64c927-plugins-config-volume\") pod \"292838de-77d4-40f1-a151-8d77bd64c927\" (UID: \"292838de-77d4-40f1-a151-8d77bd64c927\") " Jul 27 16:48:58.423873 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:58.423761 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b93bd29d-fb81-447e-b61e-0909af95968e-plugins-config-volume\") pod \"b93bd29d-fb81-447e-b61e-0909af95968e\" (UID: \"b93bd29d-fb81-447e-b61e-0909af95968e\") " Jul 27 16:48:58.423873 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:58.423800 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpvkd\" (UniqueName: \"kubernetes.io/projected/b93bd29d-fb81-447e-b61e-0909af95968e-kube-api-access-dpvkd\") pod \"b93bd29d-fb81-447e-b61e-0909af95968e\" (UID: \"b93bd29d-fb81-447e-b61e-0909af95968e\") " Jul 27 16:48:58.423873 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:58.423836 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzr4k\" (UniqueName: \"kubernetes.io/projected/292838de-77d4-40f1-a151-8d77bd64c927-kube-api-access-bzr4k\") pod \"292838de-77d4-40f1-a151-8d77bd64c927\" (UID: \"292838de-77d4-40f1-a151-8d77bd64c927\") " Jul 27 16:48:58.424223 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:58.424175 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292838de-77d4-40f1-a151-8d77bd64c927-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "292838de-77d4-40f1-a151-8d77bd64c927" (UID: "292838de-77d4-40f1-a151-8d77bd64c927"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:48:58.424347 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:58.424230 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93bd29d-fb81-447e-b61e-0909af95968e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b93bd29d-fb81-447e-b61e-0909af95968e" (UID: "b93bd29d-fb81-447e-b61e-0909af95968e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:48:58.426237 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:58.426210 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93bd29d-fb81-447e-b61e-0909af95968e-kube-api-access-dpvkd" (OuterVolumeSpecName: "kube-api-access-dpvkd") pod "b93bd29d-fb81-447e-b61e-0909af95968e" (UID: "b93bd29d-fb81-447e-b61e-0909af95968e"). InnerVolumeSpecName "kube-api-access-dpvkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:48:58.426407 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:58.426384 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/292838de-77d4-40f1-a151-8d77bd64c927-kube-api-access-bzr4k" (OuterVolumeSpecName: "kube-api-access-bzr4k") pod "292838de-77d4-40f1-a151-8d77bd64c927" (UID: "292838de-77d4-40f1-a151-8d77bd64c927"). InnerVolumeSpecName "kube-api-access-bzr4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:48:58.525394 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:58.525358 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpvkd\" (UniqueName: \"kubernetes.io/projected/b93bd29d-fb81-447e-b61e-0909af95968e-kube-api-access-dpvkd\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:48:58.525394 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:58.525389 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzr4k\" (UniqueName: \"kubernetes.io/projected/292838de-77d4-40f1-a151-8d77bd64c927-kube-api-access-bzr4k\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:48:58.525394 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:58.525398 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292838de-77d4-40f1-a151-8d77bd64c927-plugins-config-volume\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:48:58.525617 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:58.525407 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b93bd29d-fb81-447e-b61e-0909af95968e-plugins-config-volume\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:48:59.206712 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.206673 2550 generic.go:356] "Generic (PLEG): container finished" podID="b93bd29d-fb81-447e-b61e-0909af95968e" containerID="321f56b0cc801ca75e9eadc45c12a416088556d78c6a8cec9e5e6d98536fee37" exitCode=0 Jul 27 16:48:59.207187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.206723 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7" Jul 27 16:48:59.207187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.206756 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7" event={"ID":"b93bd29d-fb81-447e-b61e-0909af95968e","Type":"ContainerDied","Data":"321f56b0cc801ca75e9eadc45c12a416088556d78c6a8cec9e5e6d98536fee37"} Jul 27 16:48:59.207187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.206811 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7" event={"ID":"b93bd29d-fb81-447e-b61e-0909af95968e","Type":"ContainerDied","Data":"dfd560293073e9432f324c218c33f6847ddb4e5e335050959354dc85d6357551"} Jul 27 16:48:59.207187 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.206840 2550 scope.go:117] "RemoveContainer" containerID="321f56b0cc801ca75e9eadc45c12a416088556d78c6a8cec9e5e6d98536fee37" Jul 27 16:48:59.208359 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.208335 2550 generic.go:356] "Generic (PLEG): container finished" podID="292838de-77d4-40f1-a151-8d77bd64c927" containerID="06ed842bc580153843fb41849c59740f5dcaea8507ba0cdd34f70cdd3cdc4bc5" exitCode=0 Jul 27 16:48:59.208478 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.208373 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr" Jul 27 16:48:59.208478 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.208377 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr" event={"ID":"292838de-77d4-40f1-a151-8d77bd64c927","Type":"ContainerDied","Data":"06ed842bc580153843fb41849c59740f5dcaea8507ba0cdd34f70cdd3cdc4bc5"} Jul 27 16:48:59.208651 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.208502 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr" event={"ID":"292838de-77d4-40f1-a151-8d77bd64c927","Type":"ContainerDied","Data":"cd5b00085b37018a58253cec1a8748fd4ee834c6839b6723593f4652cbf01219"} Jul 27 16:48:59.217991 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.217862 2550 scope.go:117] "RemoveContainer" containerID="321f56b0cc801ca75e9eadc45c12a416088556d78c6a8cec9e5e6d98536fee37" Jul 27 16:48:59.218275 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:48:59.218249 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321f56b0cc801ca75e9eadc45c12a416088556d78c6a8cec9e5e6d98536fee37\": container with ID starting with 321f56b0cc801ca75e9eadc45c12a416088556d78c6a8cec9e5e6d98536fee37 not found: ID does not exist" containerID="321f56b0cc801ca75e9eadc45c12a416088556d78c6a8cec9e5e6d98536fee37" Jul 27 16:48:59.218373 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.218286 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321f56b0cc801ca75e9eadc45c12a416088556d78c6a8cec9e5e6d98536fee37"} err="failed to get container status \"321f56b0cc801ca75e9eadc45c12a416088556d78c6a8cec9e5e6d98536fee37\": rpc error: code = NotFound desc = could not find container \"321f56b0cc801ca75e9eadc45c12a416088556d78c6a8cec9e5e6d98536fee37\": container with ID starting with 321f56b0cc801ca75e9eadc45c12a416088556d78c6a8cec9e5e6d98536fee37 not found: ID does not exist" Jul 27 16:48:59.218373 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.218304 2550 scope.go:117] "RemoveContainer" containerID="06ed842bc580153843fb41849c59740f5dcaea8507ba0cdd34f70cdd3cdc4bc5" Jul 27 16:48:59.229958 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.229887 2550 scope.go:117] "RemoveContainer" containerID="06ed842bc580153843fb41849c59740f5dcaea8507ba0cdd34f70cdd3cdc4bc5" Jul 27 16:48:59.230572 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:48:59.230543 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ed842bc580153843fb41849c59740f5dcaea8507ba0cdd34f70cdd3cdc4bc5\": container with ID starting with 06ed842bc580153843fb41849c59740f5dcaea8507ba0cdd34f70cdd3cdc4bc5 not found: ID does not exist" containerID="06ed842bc580153843fb41849c59740f5dcaea8507ba0cdd34f70cdd3cdc4bc5" Jul 27 16:48:59.230676 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.230580 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ed842bc580153843fb41849c59740f5dcaea8507ba0cdd34f70cdd3cdc4bc5"} err="failed to get container status \"06ed842bc580153843fb41849c59740f5dcaea8507ba0cdd34f70cdd3cdc4bc5\": rpc error: code = NotFound desc = could not find container \"06ed842bc580153843fb41849c59740f5dcaea8507ba0cdd34f70cdd3cdc4bc5\": container with ID starting with 06ed842bc580153843fb41849c59740f5dcaea8507ba0cdd34f70cdd3cdc4bc5 not found: ID does not exist" Jul 27 16:48:59.233589 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.233561 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7"] Jul 27 16:48:59.237481 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.237457 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-370eeced-754f9f7cd-qfvx7"] Jul 27 16:48:59.248150 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.248124 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr"] Jul 27 16:48:59.251751 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.251723 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-370eeced-85697b9556-44lcr"] Jul 27 16:48:59.851209 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.851174 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="292838de-77d4-40f1-a151-8d77bd64c927" path="/var/lib/kubelet/pods/292838de-77d4-40f1-a151-8d77bd64c927/volumes" Jul 27 16:48:59.851546 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:48:59.851531 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b93bd29d-fb81-447e-b61e-0909af95968e" path="/var/lib/kubelet/pods/b93bd29d-fb81-447e-b61e-0909af95968e/volumes" Jul 27 16:49:49.700101 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.700056 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-6ds92"] Jul 27 16:49:49.700771 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.700749 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="292838de-77d4-40f1-a151-8d77bd64c927" containerName="payload-pre-processing" Jul 27 16:49:49.700857 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.700774 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="292838de-77d4-40f1-a151-8d77bd64c927" containerName="payload-pre-processing" Jul 27 16:49:49.700857 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.700790 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b93bd29d-fb81-447e-b61e-0909af95968e" containerName="payload-processing" Jul 27 16:49:49.700857 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.700798 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b93bd29d-fb81-447e-b61e-0909af95968e" containerName="payload-processing" Jul 27 16:49:49.701100 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.700963 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="292838de-77d4-40f1-a151-8d77bd64c927" containerName="payload-pre-processing" Jul 27 16:49:49.701100 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.700979 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b93bd29d-fb81-447e-b61e-0909af95968e" containerName="payload-processing" Jul 27 16:49:49.704494 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.704471 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" Jul 27 16:49:49.707701 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.707682 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-sysctl-allowlist\"" Jul 27 16:49:49.831374 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.831340 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg2bm\" (UniqueName: \"kubernetes.io/projected/acb6534c-d91e-49b3-89e5-2d3aaa956978-kube-api-access-fg2bm\") pod \"cni-sysctl-allowlist-ds-6ds92\" (UID: \"acb6534c-d91e-49b3-89e5-2d3aaa956978\") " pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" Jul 27 16:49:49.831541 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.831393 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/acb6534c-d91e-49b3-89e5-2d3aaa956978-ready\") pod \"cni-sysctl-allowlist-ds-6ds92\" (UID: \"acb6534c-d91e-49b3-89e5-2d3aaa956978\") " pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" Jul 27 16:49:49.831541 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.831448 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acb6534c-d91e-49b3-89e5-2d3aaa956978-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-6ds92\" (UID: \"acb6534c-d91e-49b3-89e5-2d3aaa956978\") " pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" Jul 27 16:49:49.831541 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.831509 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acb6534c-d91e-49b3-89e5-2d3aaa956978-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-6ds92\" (UID: \"acb6534c-d91e-49b3-89e5-2d3aaa956978\") " pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" Jul 27 16:49:49.932348 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.932309 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/acb6534c-d91e-49b3-89e5-2d3aaa956978-ready\") pod \"cni-sysctl-allowlist-ds-6ds92\" (UID: \"acb6534c-d91e-49b3-89e5-2d3aaa956978\") " pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" Jul 27 16:49:49.932563 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.932365 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acb6534c-d91e-49b3-89e5-2d3aaa956978-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-6ds92\" (UID: \"acb6534c-d91e-49b3-89e5-2d3aaa956978\") " pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" Jul 27 16:49:49.932563 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.932468 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acb6534c-d91e-49b3-89e5-2d3aaa956978-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-6ds92\" (UID: \"acb6534c-d91e-49b3-89e5-2d3aaa956978\") " pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" Jul 27 16:49:49.932563 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.932507 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acb6534c-d91e-49b3-89e5-2d3aaa956978-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-6ds92\" (UID: \"acb6534c-d91e-49b3-89e5-2d3aaa956978\") " pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" Jul 27 16:49:49.932752 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.932680 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg2bm\" (UniqueName: \"kubernetes.io/projected/acb6534c-d91e-49b3-89e5-2d3aaa956978-kube-api-access-fg2bm\") pod \"cni-sysctl-allowlist-ds-6ds92\" (UID: \"acb6534c-d91e-49b3-89e5-2d3aaa956978\") " pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" Jul 27 16:49:49.932752 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.932722 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/acb6534c-d91e-49b3-89e5-2d3aaa956978-ready\") pod \"cni-sysctl-allowlist-ds-6ds92\" (UID: \"acb6534c-d91e-49b3-89e5-2d3aaa956978\") " pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" Jul 27 16:49:49.933088 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.933067 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acb6534c-d91e-49b3-89e5-2d3aaa956978-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-6ds92\" (UID: \"acb6534c-d91e-49b3-89e5-2d3aaa956978\") " pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" Jul 27 16:49:49.943884 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:49.943862 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg2bm\" (UniqueName: \"kubernetes.io/projected/acb6534c-d91e-49b3-89e5-2d3aaa956978-kube-api-access-fg2bm\") pod \"cni-sysctl-allowlist-ds-6ds92\" (UID: \"acb6534c-d91e-49b3-89e5-2d3aaa956978\") " pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" Jul 27 16:49:50.014369 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:50.014341 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" Jul 27 16:49:50.028049 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:49:50.028019 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb6534c_d91e_49b3_89e5_2d3aaa956978.slice/crio-70e5c081efc8fa0901f9c74cf59b64214056a73636f32fd3df95fa4d0342f149 WatchSource:0}: Error finding container 70e5c081efc8fa0901f9c74cf59b64214056a73636f32fd3df95fa4d0342f149: Status 404 returned error can't find the container with id 70e5c081efc8fa0901f9c74cf59b64214056a73636f32fd3df95fa4d0342f149 Jul 27 16:49:50.540446 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:50.540414 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" event={"ID":"acb6534c-d91e-49b3-89e5-2d3aaa956978","Type":"ContainerStarted","Data":"ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6"} Jul 27 16:49:50.540600 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:50.540467 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" event={"ID":"acb6534c-d91e-49b3-89e5-2d3aaa956978","Type":"ContainerStarted","Data":"70e5c081efc8fa0901f9c74cf59b64214056a73636f32fd3df95fa4d0342f149"} Jul 27 16:49:50.540650 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:50.540614 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" Jul 27 16:49:50.564091 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:50.564039 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" podStartSLOduration=1.5640219050000002 podStartE2EDuration="1.564021905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:49:50.561245428 +0000 UTC m=+1969.233779927" watchObservedRunningTime="2026-07-27 16:49:50.564021905 +0000 UTC m=+1969.236556404" Jul 27 16:49:51.557918 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:51.557871 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" Jul 27 16:49:52.699149 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:52.699066 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-6ds92"] Jul 27 16:49:53.559205 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:49:53.559160 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" podUID="acb6534c-d91e-49b3-89e5-2d3aaa956978" containerName="kube-multus-additional-cni-plugins" containerID="cri-o://ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6" gracePeriod=30 Jul 27 16:50:01.548925 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:50:01.548861 2550 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6" cmd=["/bin/bash","-c","test -f /ready/ready"] Jul 27 16:50:01.549777 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:50:01.549749 2550 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6" cmd=["/bin/bash","-c","test -f /ready/ready"] Jul 27 16:50:01.550656 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:50:01.550627 2550 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6" cmd=["/bin/bash","-c","test -f /ready/ready"] Jul 27 16:50:01.550755 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:50:01.550666 2550 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" podUID="acb6534c-d91e-49b3-89e5-2d3aaa956978" containerName="kube-multus-additional-cni-plugins" probeResult="unknown" Jul 27 16:50:11.548860 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:50:11.548819 2550 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6" cmd=["/bin/bash","-c","test -f /ready/ready"] Jul 27 16:50:11.551084 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:50:11.549784 2550 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6" cmd=["/bin/bash","-c","test -f /ready/ready"] Jul 27 16:50:11.551084 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:50:11.550646 2550 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6" cmd=["/bin/bash","-c","test -f /ready/ready"] Jul 27 16:50:11.551084 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:50:11.550676 2550 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" podUID="acb6534c-d91e-49b3-89e5-2d3aaa956978" containerName="kube-multus-additional-cni-plugins" probeResult="unknown" Jul 27 16:50:21.548831 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:50:21.548785 2550 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6" cmd=["/bin/bash","-c","test -f /ready/ready"] Jul 27 16:50:21.549761 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:50:21.549736 2550 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6" cmd=["/bin/bash","-c","test -f /ready/ready"] Jul 27 16:50:21.550635 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:50:21.550605 2550 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6" cmd=["/bin/bash","-c","test -f /ready/ready"] Jul 27 16:50:21.550703 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:50:21.550648 2550 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" podUID="acb6534c-d91e-49b3-89e5-2d3aaa956978" containerName="kube-multus-additional-cni-plugins" probeResult="unknown" Jul 27 16:50:23.654804 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.654778 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-6ds92_acb6534c-d91e-49b3-89e5-2d3aaa956978/kube-multus-additional-cni-plugins/0.log" Jul 27 16:50:23.657016 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.656999 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" Jul 27 16:50:23.743022 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.742996 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-6ds92_acb6534c-d91e-49b3-89e5-2d3aaa956978/kube-multus-additional-cni-plugins/0.log" Jul 27 16:50:23.743186 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.743034 2550 generic.go:356] "Generic (PLEG): container finished" podID="acb6534c-d91e-49b3-89e5-2d3aaa956978" containerID="ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6" exitCode=137 Jul 27 16:50:23.743186 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.743064 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" event={"ID":"acb6534c-d91e-49b3-89e5-2d3aaa956978","Type":"ContainerDied","Data":"ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6"} Jul 27 16:50:23.743186 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.743088 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" Jul 27 16:50:23.743186 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.743139 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-6ds92" event={"ID":"acb6534c-d91e-49b3-89e5-2d3aaa956978","Type":"ContainerDied","Data":"70e5c081efc8fa0901f9c74cf59b64214056a73636f32fd3df95fa4d0342f149"} Jul 27 16:50:23.743186 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.743161 2550 scope.go:117] "RemoveContainer" containerID="ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6" Jul 27 16:50:23.753153 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.753130 2550 scope.go:117] "RemoveContainer" containerID="ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6" Jul 27 16:50:23.753440 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:50:23.753421 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6\": container with ID starting with ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6 not found: ID does not exist" containerID="ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6" Jul 27 16:50:23.753497 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.753450 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6"} err="failed to get container status \"ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6\": rpc error: code = NotFound desc = could not find container \"ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6\": container with ID starting with ae0ff630cdd36be0a8fd22bff170db08b19573bb9783f079a35b38779c70c0d6 not found: ID does not exist" Jul 27 16:50:23.762356 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.762333 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fg2bm\" (UniqueName: \"kubernetes.io/projected/acb6534c-d91e-49b3-89e5-2d3aaa956978-kube-api-access-fg2bm\") pod \"acb6534c-d91e-49b3-89e5-2d3aaa956978\" (UID: \"acb6534c-d91e-49b3-89e5-2d3aaa956978\") " Jul 27 16:50:23.762478 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.762376 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/acb6534c-d91e-49b3-89e5-2d3aaa956978-ready\") pod \"acb6534c-d91e-49b3-89e5-2d3aaa956978\" (UID: \"acb6534c-d91e-49b3-89e5-2d3aaa956978\") " Jul 27 16:50:23.762478 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.762409 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acb6534c-d91e-49b3-89e5-2d3aaa956978-cni-sysctl-allowlist\") pod \"acb6534c-d91e-49b3-89e5-2d3aaa956978\" (UID: \"acb6534c-d91e-49b3-89e5-2d3aaa956978\") " Jul 27 16:50:23.762478 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.762473 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acb6534c-d91e-49b3-89e5-2d3aaa956978-tuning-conf-dir\") pod \"acb6534c-d91e-49b3-89e5-2d3aaa956978\" (UID: \"acb6534c-d91e-49b3-89e5-2d3aaa956978\") " Jul 27 16:50:23.762649 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.762553 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/acb6534c-d91e-49b3-89e5-2d3aaa956978-tuning-conf-dir" (OuterVolumeSpecName: "tuning-conf-dir") pod "acb6534c-d91e-49b3-89e5-2d3aaa956978" (UID: "acb6534c-d91e-49b3-89e5-2d3aaa956978"). InnerVolumeSpecName "tuning-conf-dir". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Jul 27 16:50:23.762739 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.762720 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acb6534c-d91e-49b3-89e5-2d3aaa956978-cni-sysctl-allowlist" (OuterVolumeSpecName: "cni-sysctl-allowlist") pod "acb6534c-d91e-49b3-89e5-2d3aaa956978" (UID: "acb6534c-d91e-49b3-89e5-2d3aaa956978"). InnerVolumeSpecName "cni-sysctl-allowlist". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:50:23.762789 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.762729 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb6534c-d91e-49b3-89e5-2d3aaa956978-ready" (OuterVolumeSpecName: "ready") pod "acb6534c-d91e-49b3-89e5-2d3aaa956978" (UID: "acb6534c-d91e-49b3-89e5-2d3aaa956978"). InnerVolumeSpecName "ready". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:50:23.762789 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.762727 2550 reconciler_common.go:299] "Volume detached for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acb6534c-d91e-49b3-89e5-2d3aaa956978-tuning-conf-dir\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:50:23.764681 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.764660 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb6534c-d91e-49b3-89e5-2d3aaa956978-kube-api-access-fg2bm" (OuterVolumeSpecName: "kube-api-access-fg2bm") pod "acb6534c-d91e-49b3-89e5-2d3aaa956978" (UID: "acb6534c-d91e-49b3-89e5-2d3aaa956978"). InnerVolumeSpecName "kube-api-access-fg2bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:50:23.863838 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.863814 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg2bm\" (UniqueName: \"kubernetes.io/projected/acb6534c-d91e-49b3-89e5-2d3aaa956978-kube-api-access-fg2bm\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:50:23.863838 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.863837 2550 reconciler_common.go:299] "Volume detached for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/acb6534c-d91e-49b3-89e5-2d3aaa956978-ready\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:50:23.864013 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:23.863848 2550 reconciler_common.go:299] "Volume detached for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acb6534c-d91e-49b3-89e5-2d3aaa956978-cni-sysctl-allowlist\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:50:24.061193 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:24.061164 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-6ds92"] Jul 27 16:50:24.065323 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:24.065297 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-6ds92"] Jul 27 16:50:25.850394 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:50:25.850363 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acb6534c-d91e-49b3-89e5-2d3aaa956978" path="/var/lib/kubelet/pods/acb6534c-d91e-49b3-89e5-2d3aaa956978/volumes" Jul 27 16:52:00.415920 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:00.415874 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fee06a22-6d276"] Jul 27 16:52:00.416357 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:00.416342 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acb6534c-d91e-49b3-89e5-2d3aaa956978" containerName="kube-multus-additional-cni-plugins" Jul 27 16:52:00.416357 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:00.416358 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="acb6534c-d91e-49b3-89e5-2d3aaa956978" containerName="kube-multus-additional-cni-plugins" Jul 27 16:52:00.416451 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:00.416442 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="acb6534c-d91e-49b3-89e5-2d3aaa956978" containerName="kube-multus-additional-cni-plugins" Jul 27 16:52:00.419933 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:00.419892 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fee06a22-6d276" Jul 27 16:52:00.422995 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:00.422976 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v2ffk\"" Jul 27 16:52:00.429661 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:00.429629 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fee06a22-6d276"] Jul 27 16:52:00.519931 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:00.519878 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68261fbb-9a00-4bb0-8a25-3c321173fa91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fee06a22-6d276\" (UID: \"68261fbb-9a00-4bb0-8a25-3c321173fa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fee06a22-6d276" Jul 27 16:52:00.621291 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:00.621254 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68261fbb-9a00-4bb0-8a25-3c321173fa91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fee06a22-6d276\" (UID: \"68261fbb-9a00-4bb0-8a25-3c321173fa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fee06a22-6d276" Jul 27 16:52:00.621893 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:00.621872 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68261fbb-9a00-4bb0-8a25-3c321173fa91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fee06a22-6d276\" (UID: \"68261fbb-9a00-4bb0-8a25-3c321173fa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fee06a22-6d276" Jul 27 16:52:00.732041 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:00.731951 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fee06a22-6d276" Jul 27 16:52:00.876662 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:00.876631 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fee06a22-6d276"] Jul 27 16:52:00.878371 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:52:00.878340 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68261fbb_9a00_4bb0_8a25_3c321173fa91.slice/crio-06486cbc810c9d272b789ce301609f1caea639d2ef675ac6f59de1ec477b5d80 WatchSource:0}: Error finding container 06486cbc810c9d272b789ce301609f1caea639d2ef675ac6f59de1ec477b5d80: Status 404 returned error can't find the container with id 06486cbc810c9d272b789ce301609f1caea639d2ef675ac6f59de1ec477b5d80 Jul 27 16:52:01.365709 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:01.365609 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fee06a22-6d276" event={"ID":"68261fbb-9a00-4bb0-8a25-3c321173fa91","Type":"ContainerStarted","Data":"fdbc7a9dbc5581ff2aef624ac0eaf08074270e233b87a35224304fb643448d3a"} Jul 27 16:52:01.365709 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:01.365681 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fee06a22-6d276" event={"ID":"68261fbb-9a00-4bb0-8a25-3c321173fa91","Type":"ContainerStarted","Data":"06486cbc810c9d272b789ce301609f1caea639d2ef675ac6f59de1ec477b5d80"} Jul 27 16:52:01.384717 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:01.384670 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fee06a22-6d276" podStartSLOduration=1.384654274 podStartE2EDuration="1.384654274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:52:01.384038072 +0000 UTC m=+2100.056572571" watchObservedRunningTime="2026-07-27 16:52:01.384654274 +0000 UTC m=+2100.057188773" Jul 27 16:52:01.976587 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:01.976555 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:52:01.985859 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:01.985838 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:52:02.373499 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:02.373459 2550 generic.go:356] "Generic (PLEG): container finished" podID="68261fbb-9a00-4bb0-8a25-3c321173fa91" containerID="fdbc7a9dbc5581ff2aef624ac0eaf08074270e233b87a35224304fb643448d3a" exitCode=0 Jul 27 16:52:02.373682 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:02.373550 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fee06a22-6d276" event={"ID":"68261fbb-9a00-4bb0-8a25-3c321173fa91","Type":"ContainerDied","Data":"fdbc7a9dbc5581ff2aef624ac0eaf08074270e233b87a35224304fb643448d3a"} Jul 27 16:52:03.516039 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:03.516012 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fee06a22-6d276" Jul 27 16:52:03.549045 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:03.549015 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68261fbb-9a00-4bb0-8a25-3c321173fa91-maas-api-service-ca\") pod \"68261fbb-9a00-4bb0-8a25-3c321173fa91\" (UID: \"68261fbb-9a00-4bb0-8a25-3c321173fa91\") " Jul 27 16:52:03.549334 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:03.549314 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68261fbb-9a00-4bb0-8a25-3c321173fa91-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "68261fbb-9a00-4bb0-8a25-3c321173fa91" (UID: "68261fbb-9a00-4bb0-8a25-3c321173fa91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:52:03.549424 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:03.549413 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68261fbb-9a00-4bb0-8a25-3c321173fa91-maas-api-service-ca\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:52:04.390387 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:04.390353 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fee06a22-6d276" Jul 27 16:52:04.390562 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:04.390385 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fee06a22-6d276" event={"ID":"68261fbb-9a00-4bb0-8a25-3c321173fa91","Type":"ContainerDied","Data":"06486cbc810c9d272b789ce301609f1caea639d2ef675ac6f59de1ec477b5d80"} Jul 27 16:52:04.390562 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:04.390434 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06486cbc810c9d272b789ce301609f1caea639d2ef675ac6f59de1ec477b5d80" Jul 27 16:52:15.516105 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:15.516074 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fee06a22-6d276"] Jul 27 16:52:15.518586 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:15.518561 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fee06a22-6d276"] Jul 27 16:52:15.851765 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:15.851685 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68261fbb-9a00-4bb0-8a25-3c321173fa91" path="/var/lib/kubelet/pods/68261fbb-9a00-4bb0-8a25-3c321173fa91/volumes" Jul 27 16:52:46.226237 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:46.226194 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n"] Jul 27 16:52:46.226971 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:46.226952 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68261fbb-9a00-4bb0-8a25-3c321173fa91" containerName="revoke-keys" Jul 27 16:52:46.227044 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:46.226978 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="68261fbb-9a00-4bb0-8a25-3c321173fa91" containerName="revoke-keys" Jul 27 16:52:46.227095 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:46.227086 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="68261fbb-9a00-4bb0-8a25-3c321173fa91" containerName="revoke-keys" Jul 27 16:52:46.230122 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:46.230103 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n" Jul 27 16:52:46.233529 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:46.233510 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-dc13869f-dockercfg-vcnjk\"" Jul 27 16:52:46.233801 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:46.233783 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-dc13869f\"" Jul 27 16:52:46.241941 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:46.241917 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n"] Jul 27 16:52:46.262638 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:46.262598 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b7fw\" (UniqueName: \"kubernetes.io/projected/76285f12-4309-4578-878f-fc45048a0a84-kube-api-access-6b7fw\") pod \"payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n\" (UID: \"76285f12-4309-4578-878f-fc45048a0a84\") " pod="openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n" Jul 27 16:52:46.262820 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:46.262642 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76285f12-4309-4578-878f-fc45048a0a84-plugins-config-volume\") pod \"payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n\" (UID: \"76285f12-4309-4578-878f-fc45048a0a84\") " pod="openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n" Jul 27 16:52:46.363803 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:46.363756 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76285f12-4309-4578-878f-fc45048a0a84-plugins-config-volume\") pod \"payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n\" (UID: \"76285f12-4309-4578-878f-fc45048a0a84\") " pod="openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n" Jul 27 16:52:46.364024 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:46.363888 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b7fw\" (UniqueName: \"kubernetes.io/projected/76285f12-4309-4578-878f-fc45048a0a84-kube-api-access-6b7fw\") pod \"payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n\" (UID: \"76285f12-4309-4578-878f-fc45048a0a84\") " pod="openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n" Jul 27 16:52:46.364590 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:46.364567 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76285f12-4309-4578-878f-fc45048a0a84-plugins-config-volume\") pod \"payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n\" (UID: \"76285f12-4309-4578-878f-fc45048a0a84\") " pod="openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n" Jul 27 16:52:46.376249 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:46.376213 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b7fw\" (UniqueName: \"kubernetes.io/projected/76285f12-4309-4578-878f-fc45048a0a84-kube-api-access-6b7fw\") pod \"payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n\" (UID: \"76285f12-4309-4578-878f-fc45048a0a84\") " pod="openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n" Jul 27 16:52:46.542274 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:46.542224 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n" Jul 27 16:52:46.694093 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:46.694051 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n"] Jul 27 16:52:46.696747 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:52:46.696716 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76285f12_4309_4578_878f_fc45048a0a84.slice/crio-4681adb0e8f098c743bf11ba2dcc821b0888faeadfa834b7787e1693cfe148df WatchSource:0}: Error finding container 4681adb0e8f098c743bf11ba2dcc821b0888faeadfa834b7787e1693cfe148df: Status 404 returned error can't find the container with id 4681adb0e8f098c743bf11ba2dcc821b0888faeadfa834b7787e1693cfe148df Jul 27 16:52:47.657700 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:47.657665 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n" event={"ID":"76285f12-4309-4578-878f-fc45048a0a84","Type":"ContainerStarted","Data":"61cc974a9fac56512bd9a11365106f339272efde978068ea9f573da83cd80e1c"} Jul 27 16:52:47.658090 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:47.657723 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n" event={"ID":"76285f12-4309-4578-878f-fc45048a0a84","Type":"ContainerStarted","Data":"4681adb0e8f098c743bf11ba2dcc821b0888faeadfa834b7787e1693cfe148df"} Jul 27 16:52:47.658090 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:47.657746 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n" Jul 27 16:52:47.677536 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:47.677484 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n" podStartSLOduration=1.677470282 podStartE2EDuration="1.677470282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:52:47.67604124 +0000 UTC m=+2146.348575740" watchObservedRunningTime="2026-07-27 16:52:47.677470282 +0000 UTC m=+2146.350004782" Jul 27 16:52:58.667779 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:52:58.667737 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n" Jul 27 16:53:02.131326 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:02.131296 2550 scope.go:117] "RemoveContainer" containerID="c733728c7ac77b71c6dfe53ef697e56f9665eeaff3fb56bc321c9493d28ba3b8" Jul 27 16:53:02.140463 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:02.140441 2550 scope.go:117] "RemoveContainer" containerID="9c1a8ca775cc7bfea0fa7efcc5d23707eed891728cfecb7004b6a28eb108382b" Jul 27 16:53:29.597531 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:29.597484 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn"] Jul 27 16:53:29.601509 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:29.601488 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn" Jul 27 16:53:29.604979 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:29.604889 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v2ffk\"" Jul 27 16:53:29.610064 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:29.610004 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn"] Jul 27 16:53:29.659297 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:29.659259 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e249f3ff-8d10-4547-9968-221a55b238ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn\" (UID: \"e249f3ff-8d10-4547-9968-221a55b238ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn" Jul 27 16:53:29.760303 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:29.760270 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e249f3ff-8d10-4547-9968-221a55b238ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn\" (UID: \"e249f3ff-8d10-4547-9968-221a55b238ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn" Jul 27 16:53:29.760958 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:29.760891 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e249f3ff-8d10-4547-9968-221a55b238ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn\" (UID: \"e249f3ff-8d10-4547-9968-221a55b238ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn" Jul 27 16:53:29.915850 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:29.915752 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn" Jul 27 16:53:30.265451 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:30.265413 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn"] Jul 27 16:53:30.266715 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:53:30.266685 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode249f3ff_8d10_4547_9968_221a55b238ee.slice/crio-f1969d3f2fead8194796bca3209e2947cf51148b4ff3981eb3237fbd3638e605 WatchSource:0}: Error finding container f1969d3f2fead8194796bca3209e2947cf51148b4ff3981eb3237fbd3638e605: Status 404 returned error can't find the container with id f1969d3f2fead8194796bca3209e2947cf51148b4ff3981eb3237fbd3638e605 Jul 27 16:53:30.926307 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:30.926274 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn" event={"ID":"e249f3ff-8d10-4547-9968-221a55b238ee","Type":"ContainerStarted","Data":"f1b23423fa81453b304b1410c1a796cb8807088d1ccdd45c37757d55e6d4c320"} Jul 27 16:53:30.926700 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:30.926363 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn" event={"ID":"e249f3ff-8d10-4547-9968-221a55b238ee","Type":"ContainerStarted","Data":"f1969d3f2fead8194796bca3209e2947cf51148b4ff3981eb3237fbd3638e605"} Jul 27 16:53:30.945743 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:30.945696 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn" podStartSLOduration=1.9456827300000001 podStartE2EDuration="1.94568273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:53:30.943303178 +0000 UTC m=+2189.615837674" watchObservedRunningTime="2026-07-27 16:53:30.94568273 +0000 UTC m=+2189.618217229" Jul 27 16:53:31.935729 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:31.935694 2550 generic.go:356] "Generic (PLEG): container finished" podID="e249f3ff-8d10-4547-9968-221a55b238ee" containerID="f1b23423fa81453b304b1410c1a796cb8807088d1ccdd45c37757d55e6d4c320" exitCode=0 Jul 27 16:53:31.936143 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:31.935783 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn" event={"ID":"e249f3ff-8d10-4547-9968-221a55b238ee","Type":"ContainerDied","Data":"f1b23423fa81453b304b1410c1a796cb8807088d1ccdd45c37757d55e6d4c320"} Jul 27 16:53:33.078774 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:33.078751 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn" Jul 27 16:53:33.189421 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:33.189392 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e249f3ff-8d10-4547-9968-221a55b238ee-maas-api-service-ca\") pod \"e249f3ff-8d10-4547-9968-221a55b238ee\" (UID: \"e249f3ff-8d10-4547-9968-221a55b238ee\") " Jul 27 16:53:33.189728 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:33.189707 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e249f3ff-8d10-4547-9968-221a55b238ee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e249f3ff-8d10-4547-9968-221a55b238ee" (UID: "e249f3ff-8d10-4547-9968-221a55b238ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:53:33.290555 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:33.290526 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e249f3ff-8d10-4547-9968-221a55b238ee-maas-api-service-ca\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:53:33.949159 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:33.949130 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn" Jul 27 16:53:33.949330 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:33.949174 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn" event={"ID":"e249f3ff-8d10-4547-9968-221a55b238ee","Type":"ContainerDied","Data":"f1969d3f2fead8194796bca3209e2947cf51148b4ff3981eb3237fbd3638e605"} Jul 27 16:53:33.949330 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:33.949219 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1969d3f2fead8194796bca3209e2947cf51148b4ff3981eb3237fbd3638e605" Jul 27 16:53:45.231479 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:45.231443 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n"] Jul 27 16:53:45.231969 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:45.231639 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n" podUID="76285f12-4309-4578-878f-fc45048a0a84" containerName="payload-processing" containerID="cri-o://61cc974a9fac56512bd9a11365106f339272efde978068ea9f573da83cd80e1c" gracePeriod=30 Jul 27 16:53:45.497615 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:45.497549 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n" Jul 27 16:53:45.616580 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:45.616530 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6b7fw\" (UniqueName: \"kubernetes.io/projected/76285f12-4309-4578-878f-fc45048a0a84-kube-api-access-6b7fw\") pod \"76285f12-4309-4578-878f-fc45048a0a84\" (UID: \"76285f12-4309-4578-878f-fc45048a0a84\") " Jul 27 16:53:45.616758 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:45.616637 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76285f12-4309-4578-878f-fc45048a0a84-plugins-config-volume\") pod \"76285f12-4309-4578-878f-fc45048a0a84\" (UID: \"76285f12-4309-4578-878f-fc45048a0a84\") " Jul 27 16:53:45.617019 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:45.616997 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76285f12-4309-4578-878f-fc45048a0a84-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "76285f12-4309-4578-878f-fc45048a0a84" (UID: "76285f12-4309-4578-878f-fc45048a0a84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:53:45.618770 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:45.618750 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76285f12-4309-4578-878f-fc45048a0a84-kube-api-access-6b7fw" (OuterVolumeSpecName: "kube-api-access-6b7fw") pod "76285f12-4309-4578-878f-fc45048a0a84" (UID: "76285f12-4309-4578-878f-fc45048a0a84"). InnerVolumeSpecName "kube-api-access-6b7fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:53:45.717483 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:45.717431 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b7fw\" (UniqueName: \"kubernetes.io/projected/76285f12-4309-4578-878f-fc45048a0a84-kube-api-access-6b7fw\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:53:45.717483 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:45.717477 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76285f12-4309-4578-878f-fc45048a0a84-plugins-config-volume\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:53:46.023863 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:46.023826 2550 generic.go:356] "Generic (PLEG): container finished" podID="76285f12-4309-4578-878f-fc45048a0a84" containerID="61cc974a9fac56512bd9a11365106f339272efde978068ea9f573da83cd80e1c" exitCode=0 Jul 27 16:53:46.024074 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:46.023870 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n" Jul 27 16:53:46.024074 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:46.023940 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n" event={"ID":"76285f12-4309-4578-878f-fc45048a0a84","Type":"ContainerDied","Data":"61cc974a9fac56512bd9a11365106f339272efde978068ea9f573da83cd80e1c"} Jul 27 16:53:46.024074 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:46.023998 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n" event={"ID":"76285f12-4309-4578-878f-fc45048a0a84","Type":"ContainerDied","Data":"4681adb0e8f098c743bf11ba2dcc821b0888faeadfa834b7787e1693cfe148df"} Jul 27 16:53:46.024074 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:46.024027 2550 scope.go:117] "RemoveContainer" containerID="61cc974a9fac56512bd9a11365106f339272efde978068ea9f573da83cd80e1c" Jul 27 16:53:46.034014 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:46.033997 2550 scope.go:117] "RemoveContainer" containerID="61cc974a9fac56512bd9a11365106f339272efde978068ea9f573da83cd80e1c" Jul 27 16:53:46.034325 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:53:46.034304 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61cc974a9fac56512bd9a11365106f339272efde978068ea9f573da83cd80e1c\": container with ID starting with 61cc974a9fac56512bd9a11365106f339272efde978068ea9f573da83cd80e1c not found: ID does not exist" containerID="61cc974a9fac56512bd9a11365106f339272efde978068ea9f573da83cd80e1c" Jul 27 16:53:46.034430 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:46.034331 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61cc974a9fac56512bd9a11365106f339272efde978068ea9f573da83cd80e1c"} err="failed to get container status \"61cc974a9fac56512bd9a11365106f339272efde978068ea9f573da83cd80e1c\": rpc error: code = NotFound desc = could not find container \"61cc974a9fac56512bd9a11365106f339272efde978068ea9f573da83cd80e1c\": container with ID starting with 61cc974a9fac56512bd9a11365106f339272efde978068ea9f573da83cd80e1c not found: ID does not exist" Jul 27 16:53:46.043865 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:46.043843 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n"] Jul 27 16:53:46.048034 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:46.048014 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dc13869f-6bb4ddc9c4-t429n"] Jul 27 16:53:47.850789 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:47.850755 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76285f12-4309-4578-878f-fc45048a0a84" path="/var/lib/kubelet/pods/76285f12-4309-4578-878f-fc45048a0a84/volumes" Jul 27 16:53:49.683598 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:49.683567 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn"] Jul 27 16:53:49.685968 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:49.685948 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc13869f-ds4tn"] Jul 27 16:53:49.851560 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:53:49.851525 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e249f3ff-8d10-4547-9968-221a55b238ee" path="/var/lib/kubelet/pods/e249f3ff-8d10-4547-9968-221a55b238ee/volumes" Jul 27 16:54:02.167837 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:54:02.167811 2550 scope.go:117] "RemoveContainer" containerID="bce5e9da35822c5af4faa4463e7f8dc18aae54ded88c1ea31c6c071ec07e0640" Jul 27 16:55:53.786165 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.786072 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq"] Jul 27 16:55:53.788391 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.786720 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e249f3ff-8d10-4547-9968-221a55b238ee" containerName="revoke-keys" Jul 27 16:55:53.788391 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.786735 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e249f3ff-8d10-4547-9968-221a55b238ee" containerName="revoke-keys" Jul 27 16:55:53.788391 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.786748 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76285f12-4309-4578-878f-fc45048a0a84" containerName="payload-processing" Jul 27 16:55:53.788391 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.786755 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="76285f12-4309-4578-878f-fc45048a0a84" containerName="payload-processing" Jul 27 16:55:53.788391 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.786828 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e249f3ff-8d10-4547-9968-221a55b238ee" containerName="revoke-keys" Jul 27 16:55:53.788391 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.786839 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="76285f12-4309-4578-878f-fc45048a0a84" containerName="payload-processing" Jul 27 16:55:53.789676 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.789657 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq" Jul 27 16:55:53.794862 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.794838 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5de735\"" Jul 27 16:55:53.795233 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.795212 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5de735-dockercfg-bkv6f\"" Jul 27 16:55:53.805485 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.805458 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq"] Jul 27 16:55:53.825097 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.825063 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j"] Jul 27 16:55:53.828515 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.828499 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j" Jul 27 16:55:53.852868 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.852838 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j"] Jul 27 16:55:53.858959 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.858938 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9dce052-a9ea-487a-b046-19bdeeb537bd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq\" (UID: \"a9dce052-a9ea-487a-b046-19bdeeb537bd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq" Jul 27 16:55:53.859092 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.859075 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bmhx\" (UniqueName: \"kubernetes.io/projected/a9dce052-a9ea-487a-b046-19bdeeb537bd-kube-api-access-8bmhx\") pod \"payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq\" (UID: \"a9dce052-a9ea-487a-b046-19bdeeb537bd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq" Jul 27 16:55:53.960167 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.960131 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4mxd\" (UniqueName: \"kubernetes.io/projected/85519221-a8f2-44d3-ada1-72559b0107c9-kube-api-access-t4mxd\") pod \"payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j\" (UID: \"85519221-a8f2-44d3-ada1-72559b0107c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j" Jul 27 16:55:53.960349 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.960205 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85519221-a8f2-44d3-ada1-72559b0107c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j\" (UID: \"85519221-a8f2-44d3-ada1-72559b0107c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j" Jul 27 16:55:53.960349 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.960237 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bmhx\" (UniqueName: \"kubernetes.io/projected/a9dce052-a9ea-487a-b046-19bdeeb537bd-kube-api-access-8bmhx\") pod \"payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq\" (UID: \"a9dce052-a9ea-487a-b046-19bdeeb537bd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq" Jul 27 16:55:53.960349 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.960268 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9dce052-a9ea-487a-b046-19bdeeb537bd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq\" (UID: \"a9dce052-a9ea-487a-b046-19bdeeb537bd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq" Jul 27 16:55:53.961003 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.960980 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9dce052-a9ea-487a-b046-19bdeeb537bd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq\" (UID: \"a9dce052-a9ea-487a-b046-19bdeeb537bd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq" Jul 27 16:55:53.970736 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:53.970716 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bmhx\" (UniqueName: \"kubernetes.io/projected/a9dce052-a9ea-487a-b046-19bdeeb537bd-kube-api-access-8bmhx\") pod \"payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq\" (UID: \"a9dce052-a9ea-487a-b046-19bdeeb537bd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq" Jul 27 16:55:54.061731 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:54.061637 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85519221-a8f2-44d3-ada1-72559b0107c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j\" (UID: \"85519221-a8f2-44d3-ada1-72559b0107c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j" Jul 27 16:55:54.061894 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:54.061759 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4mxd\" (UniqueName: \"kubernetes.io/projected/85519221-a8f2-44d3-ada1-72559b0107c9-kube-api-access-t4mxd\") pod \"payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j\" (UID: \"85519221-a8f2-44d3-ada1-72559b0107c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j" Jul 27 16:55:54.062372 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:54.062352 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85519221-a8f2-44d3-ada1-72559b0107c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j\" (UID: \"85519221-a8f2-44d3-ada1-72559b0107c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j" Jul 27 16:55:54.073683 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:54.073652 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4mxd\" (UniqueName: \"kubernetes.io/projected/85519221-a8f2-44d3-ada1-72559b0107c9-kube-api-access-t4mxd\") pod \"payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j\" (UID: \"85519221-a8f2-44d3-ada1-72559b0107c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j" Jul 27 16:55:54.103320 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:54.103285 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq" Jul 27 16:55:54.139834 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:54.139801 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j" Jul 27 16:55:54.464532 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:54.464502 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq"] Jul 27 16:55:54.465304 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:55:54.465282 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9dce052_a9ea_487a_b046_19bdeeb537bd.slice/crio-1e013f2b7ee34158d289851aa9b566a4ae55d0c7387355785308153351339aa5 WatchSource:0}: Error finding container 1e013f2b7ee34158d289851aa9b566a4ae55d0c7387355785308153351339aa5: Status 404 returned error can't find the container with id 1e013f2b7ee34158d289851aa9b566a4ae55d0c7387355785308153351339aa5 Jul 27 16:55:54.509545 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:54.509515 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j"] Jul 27 16:55:54.519894 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:55:54.519872 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85519221_a8f2_44d3_ada1_72559b0107c9.slice/crio-15881cca1ba9fd9a530307a44fbe95d3de72378ed631890231e76be30f01fd7e WatchSource:0}: Error finding container 15881cca1ba9fd9a530307a44fbe95d3de72378ed631890231e76be30f01fd7e: Status 404 returned error can't find the container with id 15881cca1ba9fd9a530307a44fbe95d3de72378ed631890231e76be30f01fd7e Jul 27 16:55:54.822996 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:54.822952 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j" event={"ID":"85519221-a8f2-44d3-ada1-72559b0107c9","Type":"ContainerStarted","Data":"5cce257a26307020fde96b66ff71ee096b872c188577bbbac6c7a2e0c5c7898f"} Jul 27 16:55:54.823455 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:54.823015 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j" event={"ID":"85519221-a8f2-44d3-ada1-72559b0107c9","Type":"ContainerStarted","Data":"15881cca1ba9fd9a530307a44fbe95d3de72378ed631890231e76be30f01fd7e"} Jul 27 16:55:54.823455 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:54.823046 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j" Jul 27 16:55:54.824603 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:54.824578 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq" event={"ID":"a9dce052-a9ea-487a-b046-19bdeeb537bd","Type":"ContainerStarted","Data":"b2890603c2ac40c9e7b4264acae99760f458ba4fadd8215ad78db518917331c6"} Jul 27 16:55:54.824695 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:54.824625 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq" event={"ID":"a9dce052-a9ea-487a-b046-19bdeeb537bd","Type":"ContainerStarted","Data":"1e013f2b7ee34158d289851aa9b566a4ae55d0c7387355785308153351339aa5"} Jul 27 16:55:54.824695 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:54.824653 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq" Jul 27 16:55:54.844122 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:54.844077 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j" podStartSLOduration=1.844059956 podStartE2EDuration="1.844059956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:55:54.840975504 +0000 UTC m=+2333.513510002" watchObservedRunningTime="2026-07-27 16:55:54.844059956 +0000 UTC m=+2333.516594457" Jul 27 16:55:54.861888 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:55:54.861837 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq" podStartSLOduration=1.861819139 podStartE2EDuration="1.861819139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:55:54.859726491 +0000 UTC m=+2333.532260991" watchObservedRunningTime="2026-07-27 16:55:54.861819139 +0000 UTC m=+2333.534353637" Jul 27 16:56:05.835620 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:56:05.835586 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j" Jul 27 16:56:05.836077 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:56:05.835753 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq" Jul 27 16:57:02.020454 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:02.020415 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:57:02.029981 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:02.029957 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 16:57:08.161996 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:08.161955 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z"] Jul 27 16:57:08.165627 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:08.165610 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z" Jul 27 16:57:08.168842 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:08.168822 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v2ffk\"" Jul 27 16:57:08.174178 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:08.173960 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z"] Jul 27 16:57:08.181930 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:08.181887 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c699322-2169-46b1-862e-a01bba952fef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z\" (UID: \"6c699322-2169-46b1-862e-a01bba952fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z" Jul 27 16:57:08.283006 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:08.282968 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c699322-2169-46b1-862e-a01bba952fef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z\" (UID: \"6c699322-2169-46b1-862e-a01bba952fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z" Jul 27 16:57:08.283614 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:08.283595 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c699322-2169-46b1-862e-a01bba952fef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z\" (UID: \"6c699322-2169-46b1-862e-a01bba952fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z" Jul 27 16:57:08.478469 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:08.478384 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z" Jul 27 16:57:08.636869 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:08.636835 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z"] Jul 27 16:57:08.638433 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:57:08.638408 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c699322_2169_46b1_862e_a01bba952fef.slice/crio-4f0be3366668a8889e5742e7296ee5f0348e6e88188f1b22d1610d52d2f32bb5 WatchSource:0}: Error finding container 4f0be3366668a8889e5742e7296ee5f0348e6e88188f1b22d1610d52d2f32bb5: Status 404 returned error can't find the container with id 4f0be3366668a8889e5742e7296ee5f0348e6e88188f1b22d1610d52d2f32bb5 Jul 27 16:57:09.310728 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:09.310693 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z" event={"ID":"6c699322-2169-46b1-862e-a01bba952fef","Type":"ContainerStarted","Data":"fc76e607f49cb14f8ed232f4679768ce5c6d496a20687c29dc49d18062f69508"} Jul 27 16:57:09.311227 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:09.311201 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z" event={"ID":"6c699322-2169-46b1-862e-a01bba952fef","Type":"ContainerStarted","Data":"4f0be3366668a8889e5742e7296ee5f0348e6e88188f1b22d1610d52d2f32bb5"} Jul 27 16:57:09.328358 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:09.328314 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z" podStartSLOduration=1.3283003039999999 podStartE2EDuration="1.328300304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:09.327457234 +0000 UTC m=+2407.999991733" watchObservedRunningTime="2026-07-27 16:57:09.328300304 +0000 UTC m=+2408.000834803" Jul 27 16:57:10.318392 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:10.318358 2550 generic.go:356] "Generic (PLEG): container finished" podID="6c699322-2169-46b1-862e-a01bba952fef" containerID="fc76e607f49cb14f8ed232f4679768ce5c6d496a20687c29dc49d18062f69508" exitCode=0 Jul 27 16:57:10.318837 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:10.318428 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z" event={"ID":"6c699322-2169-46b1-862e-a01bba952fef","Type":"ContainerDied","Data":"fc76e607f49cb14f8ed232f4679768ce5c6d496a20687c29dc49d18062f69508"} Jul 27 16:57:11.466805 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:11.466780 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z" Jul 27 16:57:11.515776 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:11.515740 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c699322-2169-46b1-862e-a01bba952fef-maas-api-service-ca\") pod \"6c699322-2169-46b1-862e-a01bba952fef\" (UID: \"6c699322-2169-46b1-862e-a01bba952fef\") " Jul 27 16:57:11.516108 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:11.516086 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c699322-2169-46b1-862e-a01bba952fef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c699322-2169-46b1-862e-a01bba952fef" (UID: "6c699322-2169-46b1-862e-a01bba952fef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:57:11.516241 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:11.516221 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c699322-2169-46b1-862e-a01bba952fef-maas-api-service-ca\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:57:12.331521 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:12.331491 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z" event={"ID":"6c699322-2169-46b1-862e-a01bba952fef","Type":"ContainerDied","Data":"4f0be3366668a8889e5742e7296ee5f0348e6e88188f1b22d1610d52d2f32bb5"} Jul 27 16:57:12.331696 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:12.331541 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f0be3366668a8889e5742e7296ee5f0348e6e88188f1b22d1610d52d2f32bb5" Jul 27 16:57:12.331696 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:12.331498 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z" Jul 27 16:57:23.251777 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:23.251703 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z"] Jul 27 16:57:23.254275 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:23.254252 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-23e152-bkz9z"] Jul 27 16:57:23.850655 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:23.850624 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c699322-2169-46b1-862e-a01bba952fef" path="/var/lib/kubelet/pods/6c699322-2169-46b1-862e-a01bba952fef/volumes" Jul 27 16:57:30.549033 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:30.548995 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5de735-77bfv"] Jul 27 16:57:30.549494 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:30.549480 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c699322-2169-46b1-862e-a01bba952fef" containerName="revoke-keys" Jul 27 16:57:30.549494 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:30.549493 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c699322-2169-46b1-862e-a01bba952fef" containerName="revoke-keys" Jul 27 16:57:30.549590 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:30.549580 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c699322-2169-46b1-862e-a01bba952fef" containerName="revoke-keys" Jul 27 16:57:30.553085 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:30.553070 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5de735-77bfv" Jul 27 16:57:30.557017 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:30.556995 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v2ffk\"" Jul 27 16:57:30.563345 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:30.563319 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5de735-77bfv"] Jul 27 16:57:30.703990 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:30.703951 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d66c88c-7bce-4bb0-b814-8fbc077fde73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5de735-77bfv\" (UID: \"3d66c88c-7bce-4bb0-b814-8fbc077fde73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5de735-77bfv" Jul 27 16:57:30.805708 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:30.805609 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d66c88c-7bce-4bb0-b814-8fbc077fde73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5de735-77bfv\" (UID: \"3d66c88c-7bce-4bb0-b814-8fbc077fde73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5de735-77bfv" Jul 27 16:57:30.806368 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:30.806343 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d66c88c-7bce-4bb0-b814-8fbc077fde73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5de735-77bfv\" (UID: \"3d66c88c-7bce-4bb0-b814-8fbc077fde73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5de735-77bfv" Jul 27 16:57:30.864709 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:30.864679 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5de735-77bfv" Jul 27 16:57:31.018322 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:31.018292 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5de735-77bfv"] Jul 27 16:57:31.019227 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:57:31.019198 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d66c88c_7bce_4bb0_b814_8fbc077fde73.slice/crio-4d606fc87bb762ca154c80e4049f462eb44cdc66d2cb193711f5212ce009f8b8 WatchSource:0}: Error finding container 4d606fc87bb762ca154c80e4049f462eb44cdc66d2cb193711f5212ce009f8b8: Status 404 returned error can't find the container with id 4d606fc87bb762ca154c80e4049f462eb44cdc66d2cb193711f5212ce009f8b8 Jul 27 16:57:31.453365 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:31.453278 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5de735-77bfv" event={"ID":"3d66c88c-7bce-4bb0-b814-8fbc077fde73","Type":"ContainerStarted","Data":"d6149ab3d014478919bf8a591b556b1adeb97f87494fdb4b9e33cb0c0e881192"} Jul 27 16:57:31.453365 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:31.453333 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5de735-77bfv" event={"ID":"3d66c88c-7bce-4bb0-b814-8fbc077fde73","Type":"ContainerStarted","Data":"4d606fc87bb762ca154c80e4049f462eb44cdc66d2cb193711f5212ce009f8b8"} Jul 27 16:57:31.476311 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:31.476261 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5de735-77bfv" podStartSLOduration=1.476245278 podStartE2EDuration="1.476245278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:31.472196825 +0000 UTC m=+2430.144731325" watchObservedRunningTime="2026-07-27 16:57:31.476245278 +0000 UTC m=+2430.148779839" Jul 27 16:57:32.461766 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:32.461728 2550 generic.go:356] "Generic (PLEG): container finished" podID="3d66c88c-7bce-4bb0-b814-8fbc077fde73" containerID="d6149ab3d014478919bf8a591b556b1adeb97f87494fdb4b9e33cb0c0e881192" exitCode=0 Jul 27 16:57:32.462248 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:32.461816 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5de735-77bfv" event={"ID":"3d66c88c-7bce-4bb0-b814-8fbc077fde73","Type":"ContainerDied","Data":"d6149ab3d014478919bf8a591b556b1adeb97f87494fdb4b9e33cb0c0e881192"} Jul 27 16:57:33.619997 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:33.619971 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5de735-77bfv" Jul 27 16:57:33.734095 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:33.733998 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d66c88c-7bce-4bb0-b814-8fbc077fde73-maas-api-service-ca\") pod \"3d66c88c-7bce-4bb0-b814-8fbc077fde73\" (UID: \"3d66c88c-7bce-4bb0-b814-8fbc077fde73\") " Jul 27 16:57:33.734398 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:33.734373 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d66c88c-7bce-4bb0-b814-8fbc077fde73-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d66c88c-7bce-4bb0-b814-8fbc077fde73" (UID: "3d66c88c-7bce-4bb0-b814-8fbc077fde73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:57:33.835512 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:33.835480 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d66c88c-7bce-4bb0-b814-8fbc077fde73-maas-api-service-ca\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:57:34.477627 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:34.477594 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5de735-77bfv" event={"ID":"3d66c88c-7bce-4bb0-b814-8fbc077fde73","Type":"ContainerDied","Data":"4d606fc87bb762ca154c80e4049f462eb44cdc66d2cb193711f5212ce009f8b8"} Jul 27 16:57:34.477627 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:34.477624 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5de735-77bfv" Jul 27 16:57:34.477844 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:34.477643 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d606fc87bb762ca154c80e4049f462eb44cdc66d2cb193711f5212ce009f8b8" Jul 27 16:57:41.033392 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.033352 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq"] Jul 27 16:57:41.033868 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.033576 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq" podUID="a9dce052-a9ea-487a-b046-19bdeeb537bd" containerName="payload-processing" containerID="cri-o://b2890603c2ac40c9e7b4264acae99760f458ba4fadd8215ad78db518917331c6" gracePeriod=30 Jul 27 16:57:41.042360 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.042018 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j"] Jul 27 16:57:41.042360 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.042268 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j" podUID="85519221-a8f2-44d3-ada1-72559b0107c9" containerName="payload-pre-processing" containerID="cri-o://5cce257a26307020fde96b66ff71ee096b872c188577bbbac6c7a2e0c5c7898f" gracePeriod=30 Jul 27 16:57:41.339473 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.339445 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq" Jul 27 16:57:41.342386 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.342365 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j" Jul 27 16:57:41.405208 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.405180 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9dce052-a9ea-487a-b046-19bdeeb537bd-plugins-config-volume\") pod \"a9dce052-a9ea-487a-b046-19bdeeb537bd\" (UID: \"a9dce052-a9ea-487a-b046-19bdeeb537bd\") " Jul 27 16:57:41.405401 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.405249 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85519221-a8f2-44d3-ada1-72559b0107c9-plugins-config-volume\") pod \"85519221-a8f2-44d3-ada1-72559b0107c9\" (UID: \"85519221-a8f2-44d3-ada1-72559b0107c9\") " Jul 27 16:57:41.405401 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.405310 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4mxd\" (UniqueName: \"kubernetes.io/projected/85519221-a8f2-44d3-ada1-72559b0107c9-kube-api-access-t4mxd\") pod \"85519221-a8f2-44d3-ada1-72559b0107c9\" (UID: \"85519221-a8f2-44d3-ada1-72559b0107c9\") " Jul 27 16:57:41.405401 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.405395 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bmhx\" (UniqueName: \"kubernetes.io/projected/a9dce052-a9ea-487a-b046-19bdeeb537bd-kube-api-access-8bmhx\") pod \"a9dce052-a9ea-487a-b046-19bdeeb537bd\" (UID: \"a9dce052-a9ea-487a-b046-19bdeeb537bd\") " Jul 27 16:57:41.405663 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.405639 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85519221-a8f2-44d3-ada1-72559b0107c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "85519221-a8f2-44d3-ada1-72559b0107c9" (UID: "85519221-a8f2-44d3-ada1-72559b0107c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:57:41.405663 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.405608 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9dce052-a9ea-487a-b046-19bdeeb537bd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a9dce052-a9ea-487a-b046-19bdeeb537bd" (UID: "a9dce052-a9ea-487a-b046-19bdeeb537bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:57:41.405808 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.405739 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9dce052-a9ea-487a-b046-19bdeeb537bd-plugins-config-volume\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:57:41.405808 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.405758 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85519221-a8f2-44d3-ada1-72559b0107c9-plugins-config-volume\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:57:41.407650 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.407619 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9dce052-a9ea-487a-b046-19bdeeb537bd-kube-api-access-8bmhx" (OuterVolumeSpecName: "kube-api-access-8bmhx") pod "a9dce052-a9ea-487a-b046-19bdeeb537bd" (UID: "a9dce052-a9ea-487a-b046-19bdeeb537bd"). InnerVolumeSpecName "kube-api-access-8bmhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:57:41.408201 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.408175 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85519221-a8f2-44d3-ada1-72559b0107c9-kube-api-access-t4mxd" (OuterVolumeSpecName: "kube-api-access-t4mxd") pod "85519221-a8f2-44d3-ada1-72559b0107c9" (UID: "85519221-a8f2-44d3-ada1-72559b0107c9"). InnerVolumeSpecName "kube-api-access-t4mxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:57:41.506692 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.506652 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4mxd\" (UniqueName: \"kubernetes.io/projected/85519221-a8f2-44d3-ada1-72559b0107c9-kube-api-access-t4mxd\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:57:41.506692 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.506692 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bmhx\" (UniqueName: \"kubernetes.io/projected/a9dce052-a9ea-487a-b046-19bdeeb537bd-kube-api-access-8bmhx\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:57:41.520639 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.520604 2550 generic.go:356] "Generic (PLEG): container finished" podID="a9dce052-a9ea-487a-b046-19bdeeb537bd" containerID="b2890603c2ac40c9e7b4264acae99760f458ba4fadd8215ad78db518917331c6" exitCode=0 Jul 27 16:57:41.520796 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.520646 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq" Jul 27 16:57:41.520796 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.520657 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq" event={"ID":"a9dce052-a9ea-487a-b046-19bdeeb537bd","Type":"ContainerDied","Data":"b2890603c2ac40c9e7b4264acae99760f458ba4fadd8215ad78db518917331c6"} Jul 27 16:57:41.520796 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.520711 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq" event={"ID":"a9dce052-a9ea-487a-b046-19bdeeb537bd","Type":"ContainerDied","Data":"1e013f2b7ee34158d289851aa9b566a4ae55d0c7387355785308153351339aa5"} Jul 27 16:57:41.520796 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.520734 2550 scope.go:117] "RemoveContainer" containerID="b2890603c2ac40c9e7b4264acae99760f458ba4fadd8215ad78db518917331c6" Jul 27 16:57:41.522034 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.522012 2550 generic.go:356] "Generic (PLEG): container finished" podID="85519221-a8f2-44d3-ada1-72559b0107c9" containerID="5cce257a26307020fde96b66ff71ee096b872c188577bbbac6c7a2e0c5c7898f" exitCode=0 Jul 27 16:57:41.522145 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.522047 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j" Jul 27 16:57:41.522145 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.522095 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j" event={"ID":"85519221-a8f2-44d3-ada1-72559b0107c9","Type":"ContainerDied","Data":"5cce257a26307020fde96b66ff71ee096b872c188577bbbac6c7a2e0c5c7898f"} Jul 27 16:57:41.522234 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.522155 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j" event={"ID":"85519221-a8f2-44d3-ada1-72559b0107c9","Type":"ContainerDied","Data":"15881cca1ba9fd9a530307a44fbe95d3de72378ed631890231e76be30f01fd7e"} Jul 27 16:57:41.531628 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.531614 2550 scope.go:117] "RemoveContainer" containerID="b2890603c2ac40c9e7b4264acae99760f458ba4fadd8215ad78db518917331c6" Jul 27 16:57:41.531988 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:57:41.531886 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2890603c2ac40c9e7b4264acae99760f458ba4fadd8215ad78db518917331c6\": container with ID starting with b2890603c2ac40c9e7b4264acae99760f458ba4fadd8215ad78db518917331c6 not found: ID does not exist" containerID="b2890603c2ac40c9e7b4264acae99760f458ba4fadd8215ad78db518917331c6" Jul 27 16:57:41.531988 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.531941 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2890603c2ac40c9e7b4264acae99760f458ba4fadd8215ad78db518917331c6"} err="failed to get container status \"b2890603c2ac40c9e7b4264acae99760f458ba4fadd8215ad78db518917331c6\": rpc error: code = NotFound desc = could not find container \"b2890603c2ac40c9e7b4264acae99760f458ba4fadd8215ad78db518917331c6\": container with ID starting with b2890603c2ac40c9e7b4264acae99760f458ba4fadd8215ad78db518917331c6 not found: ID does not exist" Jul 27 16:57:41.531988 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.531958 2550 scope.go:117] "RemoveContainer" containerID="5cce257a26307020fde96b66ff71ee096b872c188577bbbac6c7a2e0c5c7898f" Jul 27 16:57:41.541157 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.541139 2550 scope.go:117] "RemoveContainer" containerID="5cce257a26307020fde96b66ff71ee096b872c188577bbbac6c7a2e0c5c7898f" Jul 27 16:57:41.541395 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:57:41.541378 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cce257a26307020fde96b66ff71ee096b872c188577bbbac6c7a2e0c5c7898f\": container with ID starting with 5cce257a26307020fde96b66ff71ee096b872c188577bbbac6c7a2e0c5c7898f not found: ID does not exist" containerID="5cce257a26307020fde96b66ff71ee096b872c188577bbbac6c7a2e0c5c7898f" Jul 27 16:57:41.541456 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.541407 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cce257a26307020fde96b66ff71ee096b872c188577bbbac6c7a2e0c5c7898f"} err="failed to get container status \"5cce257a26307020fde96b66ff71ee096b872c188577bbbac6c7a2e0c5c7898f\": rpc error: code = NotFound desc = could not find container \"5cce257a26307020fde96b66ff71ee096b872c188577bbbac6c7a2e0c5c7898f\": container with ID starting with 5cce257a26307020fde96b66ff71ee096b872c188577bbbac6c7a2e0c5c7898f not found: ID does not exist" Jul 27 16:57:41.550635 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.550609 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j"] Jul 27 16:57:41.557435 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.557416 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5de735-56b9b568c6-qzc4j"] Jul 27 16:57:41.570480 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.570415 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq"] Jul 27 16:57:41.580324 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.580303 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5de735-56cd5bbc8b-lb4fq"] Jul 27 16:57:41.852146 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.852051 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85519221-a8f2-44d3-ada1-72559b0107c9" path="/var/lib/kubelet/pods/85519221-a8f2-44d3-ada1-72559b0107c9/volumes" Jul 27 16:57:41.852535 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:41.852514 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9dce052-a9ea-487a-b046-19bdeeb537bd" path="/var/lib/kubelet/pods/a9dce052-a9ea-487a-b046-19bdeeb537bd/volumes" Jul 27 16:57:45.628513 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:45.628464 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5de735-77bfv"] Jul 27 16:57:45.632020 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:45.631990 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5de735-77bfv"] Jul 27 16:57:45.853339 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:45.853303 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d66c88c-7bce-4bb0-b814-8fbc077fde73" path="/var/lib/kubelet/pods/3d66c88c-7bce-4bb0-b814-8fbc077fde73/volumes" Jul 27 16:57:58.015848 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.015813 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs"] Jul 27 16:57:58.016351 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.016334 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9dce052-a9ea-487a-b046-19bdeeb537bd" containerName="payload-processing" Jul 27 16:57:58.016395 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.016352 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9dce052-a9ea-487a-b046-19bdeeb537bd" containerName="payload-processing" Jul 27 16:57:58.016395 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.016362 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85519221-a8f2-44d3-ada1-72559b0107c9" containerName="payload-pre-processing" Jul 27 16:57:58.016395 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.016367 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="85519221-a8f2-44d3-ada1-72559b0107c9" containerName="payload-pre-processing" Jul 27 16:57:58.016395 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.016384 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d66c88c-7bce-4bb0-b814-8fbc077fde73" containerName="revoke-keys" Jul 27 16:57:58.016395 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.016390 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d66c88c-7bce-4bb0-b814-8fbc077fde73" containerName="revoke-keys" Jul 27 16:57:58.016554 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.016463 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="85519221-a8f2-44d3-ada1-72559b0107c9" containerName="payload-pre-processing" Jul 27 16:57:58.016554 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.016472 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9dce052-a9ea-487a-b046-19bdeeb537bd" containerName="payload-processing" Jul 27 16:57:58.016554 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.016484 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d66c88c-7bce-4bb0-b814-8fbc077fde73" containerName="revoke-keys" Jul 27 16:57:58.021193 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.021170 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs" Jul 27 16:57:58.025339 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.025315 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6dee51-dockercfg-5f2cd\"" Jul 27 16:57:58.025687 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.025668 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6dee51\"" Jul 27 16:57:58.036057 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.036036 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs"] Jul 27 16:57:58.172182 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.172145 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b698113-04f2-4e76-a4d4-38880cd94fe4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs\" (UID: \"8b698113-04f2-4e76-a4d4-38880cd94fe4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs" Jul 27 16:57:58.172355 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.172204 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpzb9\" (UniqueName: \"kubernetes.io/projected/8b698113-04f2-4e76-a4d4-38880cd94fe4-kube-api-access-vpzb9\") pod \"payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs\" (UID: \"8b698113-04f2-4e76-a4d4-38880cd94fe4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs" Jul 27 16:57:58.273086 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.272992 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b698113-04f2-4e76-a4d4-38880cd94fe4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs\" (UID: \"8b698113-04f2-4e76-a4d4-38880cd94fe4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs" Jul 27 16:57:58.273086 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.273050 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpzb9\" (UniqueName: \"kubernetes.io/projected/8b698113-04f2-4e76-a4d4-38880cd94fe4-kube-api-access-vpzb9\") pod \"payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs\" (UID: \"8b698113-04f2-4e76-a4d4-38880cd94fe4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs" Jul 27 16:57:58.273743 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.273721 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b698113-04f2-4e76-a4d4-38880cd94fe4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs\" (UID: \"8b698113-04f2-4e76-a4d4-38880cd94fe4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs" Jul 27 16:57:58.284400 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.284371 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpzb9\" (UniqueName: \"kubernetes.io/projected/8b698113-04f2-4e76-a4d4-38880cd94fe4-kube-api-access-vpzb9\") pod \"payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs\" (UID: \"8b698113-04f2-4e76-a4d4-38880cd94fe4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs" Jul 27 16:57:58.332202 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.332166 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs" Jul 27 16:57:58.509004 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.508979 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs"] Jul 27 16:57:58.510994 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:57:58.510956 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b698113_04f2_4e76_a4d4_38880cd94fe4.slice/crio-a852088b9de07f26dac40e37a8ba9647fb2a82e36a93f46775b938e12995e7ab WatchSource:0}: Error finding container a852088b9de07f26dac40e37a8ba9647fb2a82e36a93f46775b938e12995e7ab: Status 404 returned error can't find the container with id a852088b9de07f26dac40e37a8ba9647fb2a82e36a93f46775b938e12995e7ab Jul 27 16:57:58.635791 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.635760 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs" event={"ID":"8b698113-04f2-4e76-a4d4-38880cd94fe4","Type":"ContainerStarted","Data":"ce7e2e5d15712bfa7c4f332b249659b10a1c4521f61349347e1ac3c2c3e6b369"} Jul 27 16:57:58.635984 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.635810 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs" event={"ID":"8b698113-04f2-4e76-a4d4-38880cd94fe4","Type":"ContainerStarted","Data":"a852088b9de07f26dac40e37a8ba9647fb2a82e36a93f46775b938e12995e7ab"} Jul 27 16:57:58.635984 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.635886 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs" Jul 27 16:57:58.657282 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:57:58.657223 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs" podStartSLOduration=1.657206374 podStartE2EDuration="1.657206374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:58.654656371 +0000 UTC m=+2457.327190870" watchObservedRunningTime="2026-07-27 16:57:58.657206374 +0000 UTC m=+2457.329740873" Jul 27 16:58:02.223845 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:02.223811 2550 scope.go:117] "RemoveContainer" containerID="fdbc7a9dbc5581ff2aef624ac0eaf08074270e233b87a35224304fb643448d3a" Jul 27 16:58:06.567552 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:06.567514 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk"] Jul 27 16:58:06.570988 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:06.570972 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk" Jul 27 16:58:06.574020 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:06.573994 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-30f382\"" Jul 27 16:58:06.574150 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:06.574105 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-30f382-dockercfg-7994d\"" Jul 27 16:58:06.582431 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:06.582406 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk"] Jul 27 16:58:06.655097 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:06.655061 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h7ml\" (UniqueName: \"kubernetes.io/projected/f459aa57-613c-420d-92de-82e294b0714f-kube-api-access-9h7ml\") pod \"payload-processing-e2e-api-b-30f382-677b556b94-zp4zk\" (UID: \"f459aa57-613c-420d-92de-82e294b0714f\") " pod="openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk" Jul 27 16:58:06.655273 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:06.655118 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f459aa57-613c-420d-92de-82e294b0714f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-30f382-677b556b94-zp4zk\" (UID: \"f459aa57-613c-420d-92de-82e294b0714f\") " pod="openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk" Jul 27 16:58:06.756562 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:06.756527 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f459aa57-613c-420d-92de-82e294b0714f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-30f382-677b556b94-zp4zk\" (UID: \"f459aa57-613c-420d-92de-82e294b0714f\") " pod="openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk" Jul 27 16:58:06.756753 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:06.756628 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h7ml\" (UniqueName: \"kubernetes.io/projected/f459aa57-613c-420d-92de-82e294b0714f-kube-api-access-9h7ml\") pod \"payload-processing-e2e-api-b-30f382-677b556b94-zp4zk\" (UID: \"f459aa57-613c-420d-92de-82e294b0714f\") " pod="openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk" Jul 27 16:58:06.757270 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:06.757252 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f459aa57-613c-420d-92de-82e294b0714f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-30f382-677b556b94-zp4zk\" (UID: \"f459aa57-613c-420d-92de-82e294b0714f\") " pod="openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk" Jul 27 16:58:06.765599 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:06.765578 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h7ml\" (UniqueName: \"kubernetes.io/projected/f459aa57-613c-420d-92de-82e294b0714f-kube-api-access-9h7ml\") pod \"payload-processing-e2e-api-b-30f382-677b556b94-zp4zk\" (UID: \"f459aa57-613c-420d-92de-82e294b0714f\") " pod="openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk" Jul 27 16:58:06.882368 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:06.882281 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk" Jul 27 16:58:07.034941 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:07.034895 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk"] Jul 27 16:58:07.036472 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:58:07.036439 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf459aa57_613c_420d_92de_82e294b0714f.slice/crio-e878cdc4b1bd2f4d59e6ff4fc7977ca222df2e2061f42bdab55f77439d992861 WatchSource:0}: Error finding container e878cdc4b1bd2f4d59e6ff4fc7977ca222df2e2061f42bdab55f77439d992861: Status 404 returned error can't find the container with id e878cdc4b1bd2f4d59e6ff4fc7977ca222df2e2061f42bdab55f77439d992861 Jul 27 16:58:07.695325 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:07.695281 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk" event={"ID":"f459aa57-613c-420d-92de-82e294b0714f","Type":"ContainerStarted","Data":"28ab78cc8481d898db96e241e74a4cdad33edb72cd6ce4fb45d4cced8baf9a95"} Jul 27 16:58:07.695964 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:07.695351 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk" event={"ID":"f459aa57-613c-420d-92de-82e294b0714f","Type":"ContainerStarted","Data":"e878cdc4b1bd2f4d59e6ff4fc7977ca222df2e2061f42bdab55f77439d992861"} Jul 27 16:58:07.695964 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:07.695383 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk" Jul 27 16:58:07.720075 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:07.720011 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk" podStartSLOduration=1.719989659 podStartE2EDuration="1.719989659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:58:07.716726928 +0000 UTC m=+2466.389261428" watchObservedRunningTime="2026-07-27 16:58:07.719989659 +0000 UTC m=+2466.392524159" Jul 27 16:58:09.645381 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:09.645349 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs" Jul 27 16:58:18.424523 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:18.424488 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6dee51-l95bp"] Jul 27 16:58:18.428729 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:18.428708 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6dee51-l95bp" Jul 27 16:58:18.434763 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:18.434739 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v2ffk\"" Jul 27 16:58:18.444805 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:18.444778 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6dee51-l95bp"] Jul 27 16:58:18.468869 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:18.468832 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6dee51-l95bp\" (UID: \"d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6dee51-l95bp" Jul 27 16:58:18.569370 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:18.569334 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6dee51-l95bp\" (UID: \"d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6dee51-l95bp" Jul 27 16:58:18.569958 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:18.569939 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6dee51-l95bp\" (UID: \"d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6dee51-l95bp" Jul 27 16:58:18.704345 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:18.704240 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk" Jul 27 16:58:18.738920 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:18.738875 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6dee51-l95bp" Jul 27 16:58:18.901416 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:18.901387 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6dee51-l95bp"] Jul 27 16:58:18.902118 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:58:18.902089 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1848bff_58b3_4d4e_956a_c9d4f3f2b0ce.slice/crio-3fb7022005b883da92acf1419fbad3165423d96428c8ebeb7496ebf2f29af39a WatchSource:0}: Error finding container 3fb7022005b883da92acf1419fbad3165423d96428c8ebeb7496ebf2f29af39a: Status 404 returned error can't find the container with id 3fb7022005b883da92acf1419fbad3165423d96428c8ebeb7496ebf2f29af39a Jul 27 16:58:19.776881 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:19.776851 2550 generic.go:356] "Generic (PLEG): container finished" podID="d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce" containerID="aef27bbd534c54900488ac977ea3fa4677412b9a7559613e8f9568677dfae1e8" exitCode=0 Jul 27 16:58:19.777245 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:19.776889 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6dee51-l95bp" event={"ID":"d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce","Type":"ContainerDied","Data":"aef27bbd534c54900488ac977ea3fa4677412b9a7559613e8f9568677dfae1e8"} Jul 27 16:58:19.777245 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:19.776955 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6dee51-l95bp" event={"ID":"d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce","Type":"ContainerStarted","Data":"3fb7022005b883da92acf1419fbad3165423d96428c8ebeb7496ebf2f29af39a"} Jul 27 16:58:20.934011 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:20.933985 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6dee51-l95bp" Jul 27 16:58:20.989169 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:20.989137 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce-maas-api-service-ca\") pod \"d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce\" (UID: \"d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce\") " Jul 27 16:58:20.989495 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:20.989471 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce" (UID: "d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:58:21.090988 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:21.090896 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce-maas-api-service-ca\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:58:21.789884 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:21.789854 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6dee51-l95bp" Jul 27 16:58:21.789884 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:21.789873 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6dee51-l95bp" event={"ID":"d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce","Type":"ContainerDied","Data":"3fb7022005b883da92acf1419fbad3165423d96428c8ebeb7496ebf2f29af39a"} Jul 27 16:58:21.790126 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:21.789944 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fb7022005b883da92acf1419fbad3165423d96428c8ebeb7496ebf2f29af39a" Jul 27 16:58:34.118949 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:34.118896 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs"] Jul 27 16:58:34.119387 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:34.119203 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs" podUID="8b698113-04f2-4e76-a4d4-38880cd94fe4" containerName="payload-pre-processing" containerID="cri-o://ce7e2e5d15712bfa7c4f332b249659b10a1c4521f61349347e1ac3c2c3e6b369" gracePeriod=30 Jul 27 16:58:34.388299 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:34.388265 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs" Jul 27 16:58:34.435176 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:34.435142 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b698113-04f2-4e76-a4d4-38880cd94fe4-plugins-config-volume\") pod \"8b698113-04f2-4e76-a4d4-38880cd94fe4\" (UID: \"8b698113-04f2-4e76-a4d4-38880cd94fe4\") " Jul 27 16:58:34.435331 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:34.435211 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpzb9\" (UniqueName: \"kubernetes.io/projected/8b698113-04f2-4e76-a4d4-38880cd94fe4-kube-api-access-vpzb9\") pod \"8b698113-04f2-4e76-a4d4-38880cd94fe4\" (UID: \"8b698113-04f2-4e76-a4d4-38880cd94fe4\") " Jul 27 16:58:34.435587 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:34.435563 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b698113-04f2-4e76-a4d4-38880cd94fe4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b698113-04f2-4e76-a4d4-38880cd94fe4" (UID: "8b698113-04f2-4e76-a4d4-38880cd94fe4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:58:34.437403 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:34.437383 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b698113-04f2-4e76-a4d4-38880cd94fe4-kube-api-access-vpzb9" (OuterVolumeSpecName: "kube-api-access-vpzb9") pod "8b698113-04f2-4e76-a4d4-38880cd94fe4" (UID: "8b698113-04f2-4e76-a4d4-38880cd94fe4"). InnerVolumeSpecName "kube-api-access-vpzb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:58:34.536274 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:34.536238 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b698113-04f2-4e76-a4d4-38880cd94fe4-plugins-config-volume\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:58:34.536274 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:34.536265 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpzb9\" (UniqueName: \"kubernetes.io/projected/8b698113-04f2-4e76-a4d4-38880cd94fe4-kube-api-access-vpzb9\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:58:34.873233 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:34.873200 2550 generic.go:356] "Generic (PLEG): container finished" podID="8b698113-04f2-4e76-a4d4-38880cd94fe4" containerID="ce7e2e5d15712bfa7c4f332b249659b10a1c4521f61349347e1ac3c2c3e6b369" exitCode=0 Jul 27 16:58:34.873424 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:34.873240 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs" Jul 27 16:58:34.873424 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:34.873253 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs" event={"ID":"8b698113-04f2-4e76-a4d4-38880cd94fe4","Type":"ContainerDied","Data":"ce7e2e5d15712bfa7c4f332b249659b10a1c4521f61349347e1ac3c2c3e6b369"} Jul 27 16:58:34.873424 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:34.873311 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs" event={"ID":"8b698113-04f2-4e76-a4d4-38880cd94fe4","Type":"ContainerDied","Data":"a852088b9de07f26dac40e37a8ba9647fb2a82e36a93f46775b938e12995e7ab"} Jul 27 16:58:34.873424 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:34.873335 2550 scope.go:117] "RemoveContainer" containerID="ce7e2e5d15712bfa7c4f332b249659b10a1c4521f61349347e1ac3c2c3e6b369" Jul 27 16:58:34.883188 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:34.883170 2550 scope.go:117] "RemoveContainer" containerID="ce7e2e5d15712bfa7c4f332b249659b10a1c4521f61349347e1ac3c2c3e6b369" Jul 27 16:58:34.883442 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:58:34.883426 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce7e2e5d15712bfa7c4f332b249659b10a1c4521f61349347e1ac3c2c3e6b369\": container with ID starting with ce7e2e5d15712bfa7c4f332b249659b10a1c4521f61349347e1ac3c2c3e6b369 not found: ID does not exist" containerID="ce7e2e5d15712bfa7c4f332b249659b10a1c4521f61349347e1ac3c2c3e6b369" Jul 27 16:58:34.883481 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:34.883466 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce7e2e5d15712bfa7c4f332b249659b10a1c4521f61349347e1ac3c2c3e6b369"} err="failed to get container status \"ce7e2e5d15712bfa7c4f332b249659b10a1c4521f61349347e1ac3c2c3e6b369\": rpc error: code = NotFound desc = could not find container \"ce7e2e5d15712bfa7c4f332b249659b10a1c4521f61349347e1ac3c2c3e6b369\": container with ID starting with ce7e2e5d15712bfa7c4f332b249659b10a1c4521f61349347e1ac3c2c3e6b369 not found: ID does not exist" Jul 27 16:58:34.898749 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:34.898720 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs"] Jul 27 16:58:34.905796 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:34.905776 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6dee51-7c78bc645b-lq2cs"] Jul 27 16:58:35.852018 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:35.851986 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b698113-04f2-4e76-a4d4-38880cd94fe4" path="/var/lib/kubelet/pods/8b698113-04f2-4e76-a4d4-38880cd94fe4/volumes" Jul 27 16:58:38.506474 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:38.506434 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6dee51-l95bp"] Jul 27 16:58:38.515617 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:38.515587 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6dee51-l95bp"] Jul 27 16:58:39.850374 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:39.850339 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce" path="/var/lib/kubelet/pods/d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce/volumes" Jul 27 16:58:45.441094 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:45.441050 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30f382-cm786"] Jul 27 16:58:45.442114 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:45.442088 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b698113-04f2-4e76-a4d4-38880cd94fe4" containerName="payload-pre-processing" Jul 27 16:58:45.442299 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:45.442284 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b698113-04f2-4e76-a4d4-38880cd94fe4" containerName="payload-pre-processing" Jul 27 16:58:45.442414 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:45.442403 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce" containerName="revoke-keys" Jul 27 16:58:45.442493 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:45.442484 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce" containerName="revoke-keys" Jul 27 16:58:45.442807 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:45.442792 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b698113-04f2-4e76-a4d4-38880cd94fe4" containerName="payload-pre-processing" Jul 27 16:58:45.442946 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:45.442934 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1848bff-58b3-4d4e-956a-c9d4f3f2b0ce" containerName="revoke-keys" Jul 27 16:58:45.448427 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:45.448404 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30f382-cm786" Jul 27 16:58:45.451490 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:45.451468 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v2ffk\"" Jul 27 16:58:45.451663 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:45.451627 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30f382-cm786"] Jul 27 16:58:45.548511 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:45.548470 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2541881-a164-40bb-9db8-f14165a9a86e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-30f382-cm786\" (UID: \"d2541881-a164-40bb-9db8-f14165a9a86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30f382-cm786" Jul 27 16:58:45.649826 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:45.649787 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2541881-a164-40bb-9db8-f14165a9a86e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-30f382-cm786\" (UID: \"d2541881-a164-40bb-9db8-f14165a9a86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30f382-cm786" Jul 27 16:58:45.650461 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:45.650442 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2541881-a164-40bb-9db8-f14165a9a86e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-30f382-cm786\" (UID: \"d2541881-a164-40bb-9db8-f14165a9a86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30f382-cm786" Jul 27 16:58:45.761185 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:45.761100 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30f382-cm786" Jul 27 16:58:45.908969 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:45.908940 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30f382-cm786"] Jul 27 16:58:45.911414 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:58:45.911384 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2541881_a164_40bb_9db8_f14165a9a86e.slice/crio-46e6f832b5edf5502ba884cbf641cea059e63a2667528ffac3c96c820a201379 WatchSource:0}: Error finding container 46e6f832b5edf5502ba884cbf641cea059e63a2667528ffac3c96c820a201379: Status 404 returned error can't find the container with id 46e6f832b5edf5502ba884cbf641cea059e63a2667528ffac3c96c820a201379 Jul 27 16:58:45.943916 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:45.943874 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30f382-cm786" event={"ID":"d2541881-a164-40bb-9db8-f14165a9a86e","Type":"ContainerStarted","Data":"46e6f832b5edf5502ba884cbf641cea059e63a2667528ffac3c96c820a201379"} Jul 27 16:58:46.951930 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:46.951811 2550 generic.go:356] "Generic (PLEG): container finished" podID="d2541881-a164-40bb-9db8-f14165a9a86e" containerID="d6438b2dfaaebddbd459d413ea0f1eca8a87f36a389a4a8a13b328be0a32347e" exitCode=0 Jul 27 16:58:46.951930 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:46.951894 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30f382-cm786" event={"ID":"d2541881-a164-40bb-9db8-f14165a9a86e","Type":"ContainerDied","Data":"d6438b2dfaaebddbd459d413ea0f1eca8a87f36a389a4a8a13b328be0a32347e"} Jul 27 16:58:48.103059 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:48.103032 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30f382-cm786" Jul 27 16:58:48.279200 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:48.279163 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2541881-a164-40bb-9db8-f14165a9a86e-maas-api-service-ca\") pod \"d2541881-a164-40bb-9db8-f14165a9a86e\" (UID: \"d2541881-a164-40bb-9db8-f14165a9a86e\") " Jul 27 16:58:48.279534 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:48.279508 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2541881-a164-40bb-9db8-f14165a9a86e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2541881-a164-40bb-9db8-f14165a9a86e" (UID: "d2541881-a164-40bb-9db8-f14165a9a86e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:58:48.380305 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:48.380267 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2541881-a164-40bb-9db8-f14165a9a86e-maas-api-service-ca\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:58:48.965532 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:48.965501 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30f382-cm786" Jul 27 16:58:48.965712 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:48.965500 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30f382-cm786" event={"ID":"d2541881-a164-40bb-9db8-f14165a9a86e","Type":"ContainerDied","Data":"46e6f832b5edf5502ba884cbf641cea059e63a2667528ffac3c96c820a201379"} Jul 27 16:58:48.965712 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:58:48.965635 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46e6f832b5edf5502ba884cbf641cea059e63a2667528ffac3c96c820a201379" Jul 27 16:59:00.991323 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:00.991278 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk"] Jul 27 16:59:00.991779 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:00.991538 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk" podUID="f459aa57-613c-420d-92de-82e294b0714f" containerName="payload-processing" containerID="cri-o://28ab78cc8481d898db96e241e74a4cdad33edb72cd6ce4fb45d4cced8baf9a95" gracePeriod=30 Jul 27 16:59:01.264221 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:01.264194 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk" Jul 27 16:59:01.301774 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:01.301736 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h7ml\" (UniqueName: \"kubernetes.io/projected/f459aa57-613c-420d-92de-82e294b0714f-kube-api-access-9h7ml\") pod \"f459aa57-613c-420d-92de-82e294b0714f\" (UID: \"f459aa57-613c-420d-92de-82e294b0714f\") " Jul 27 16:59:01.301978 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:01.301825 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f459aa57-613c-420d-92de-82e294b0714f-plugins-config-volume\") pod \"f459aa57-613c-420d-92de-82e294b0714f\" (UID: \"f459aa57-613c-420d-92de-82e294b0714f\") " Jul 27 16:59:01.302272 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:01.302244 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f459aa57-613c-420d-92de-82e294b0714f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f459aa57-613c-420d-92de-82e294b0714f" (UID: "f459aa57-613c-420d-92de-82e294b0714f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:59:01.304257 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:01.304227 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f459aa57-613c-420d-92de-82e294b0714f-kube-api-access-9h7ml" (OuterVolumeSpecName: "kube-api-access-9h7ml") pod "f459aa57-613c-420d-92de-82e294b0714f" (UID: "f459aa57-613c-420d-92de-82e294b0714f"). InnerVolumeSpecName "kube-api-access-9h7ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:59:01.403642 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:01.403601 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h7ml\" (UniqueName: \"kubernetes.io/projected/f459aa57-613c-420d-92de-82e294b0714f-kube-api-access-9h7ml\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:59:01.403642 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:01.403640 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f459aa57-613c-420d-92de-82e294b0714f-plugins-config-volume\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 16:59:02.055411 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:02.055379 2550 generic.go:356] "Generic (PLEG): container finished" podID="f459aa57-613c-420d-92de-82e294b0714f" containerID="28ab78cc8481d898db96e241e74a4cdad33edb72cd6ce4fb45d4cced8baf9a95" exitCode=0 Jul 27 16:59:02.055876 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:02.055438 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk" Jul 27 16:59:02.055876 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:02.055459 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk" event={"ID":"f459aa57-613c-420d-92de-82e294b0714f","Type":"ContainerDied","Data":"28ab78cc8481d898db96e241e74a4cdad33edb72cd6ce4fb45d4cced8baf9a95"} Jul 27 16:59:02.055876 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:02.055530 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk" event={"ID":"f459aa57-613c-420d-92de-82e294b0714f","Type":"ContainerDied","Data":"e878cdc4b1bd2f4d59e6ff4fc7977ca222df2e2061f42bdab55f77439d992861"} Jul 27 16:59:02.055876 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:02.055552 2550 scope.go:117] "RemoveContainer" containerID="28ab78cc8481d898db96e241e74a4cdad33edb72cd6ce4fb45d4cced8baf9a95" Jul 27 16:59:02.065232 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:02.065203 2550 scope.go:117] "RemoveContainer" containerID="28ab78cc8481d898db96e241e74a4cdad33edb72cd6ce4fb45d4cced8baf9a95" Jul 27 16:59:02.065580 ip-10-0-135-70 kubenswrapper[2550]: E0727 16:59:02.065549 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ab78cc8481d898db96e241e74a4cdad33edb72cd6ce4fb45d4cced8baf9a95\": container with ID starting with 28ab78cc8481d898db96e241e74a4cdad33edb72cd6ce4fb45d4cced8baf9a95 not found: ID does not exist" containerID="28ab78cc8481d898db96e241e74a4cdad33edb72cd6ce4fb45d4cced8baf9a95" Jul 27 16:59:02.065667 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:02.065603 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ab78cc8481d898db96e241e74a4cdad33edb72cd6ce4fb45d4cced8baf9a95"} err="failed to get container status \"28ab78cc8481d898db96e241e74a4cdad33edb72cd6ce4fb45d4cced8baf9a95\": rpc error: code = NotFound desc = could not find container \"28ab78cc8481d898db96e241e74a4cdad33edb72cd6ce4fb45d4cced8baf9a95\": container with ID starting with 28ab78cc8481d898db96e241e74a4cdad33edb72cd6ce4fb45d4cced8baf9a95 not found: ID does not exist" Jul 27 16:59:02.077743 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:02.077718 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk"] Jul 27 16:59:02.081541 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:02.081509 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30f382-677b556b94-zp4zk"] Jul 27 16:59:03.850843 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:03.850810 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f459aa57-613c-420d-92de-82e294b0714f" path="/var/lib/kubelet/pods/f459aa57-613c-420d-92de-82e294b0714f/volumes" Jul 27 16:59:05.545894 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:05.545860 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30f382-cm786"] Jul 27 16:59:05.548724 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:05.548699 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30f382-cm786"] Jul 27 16:59:05.851863 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:05.851780 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2541881-a164-40bb-9db8-f14165a9a86e" path="/var/lib/kubelet/pods/d2541881-a164-40bb-9db8-f14165a9a86e/volumes" Jul 27 16:59:17.111340 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:17.111304 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7"] Jul 27 16:59:17.112010 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:17.111987 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2541881-a164-40bb-9db8-f14165a9a86e" containerName="revoke-keys" Jul 27 16:59:17.112089 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:17.112013 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2541881-a164-40bb-9db8-f14165a9a86e" containerName="revoke-keys" Jul 27 16:59:17.112089 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:17.112041 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f459aa57-613c-420d-92de-82e294b0714f" containerName="payload-processing" Jul 27 16:59:17.112089 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:17.112050 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f459aa57-613c-420d-92de-82e294b0714f" containerName="payload-processing" Jul 27 16:59:17.112188 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:17.112175 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2541881-a164-40bb-9db8-f14165a9a86e" containerName="revoke-keys" Jul 27 16:59:17.112222 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:17.112190 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f459aa57-613c-420d-92de-82e294b0714f" containerName="payload-processing" Jul 27 16:59:17.115814 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:17.115797 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7" Jul 27 16:59:17.123664 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:17.123641 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-f1b699-dockercfg-jx5wk\"" Jul 27 16:59:17.123781 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:17.123706 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-f1b699\"" Jul 27 16:59:17.129616 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:17.129595 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7"] Jul 27 16:59:17.263454 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:17.263416 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcfxl\" (UniqueName: \"kubernetes.io/projected/9549a682-fb22-422c-bae3-1dff76312415-kube-api-access-rcfxl\") pod \"payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7\" (UID: \"9549a682-fb22-422c-bae3-1dff76312415\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7" Jul 27 16:59:17.263633 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:17.263582 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9549a682-fb22-422c-bae3-1dff76312415-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7\" (UID: \"9549a682-fb22-422c-bae3-1dff76312415\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7" Jul 27 16:59:17.364338 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:17.364239 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9549a682-fb22-422c-bae3-1dff76312415-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7\" (UID: \"9549a682-fb22-422c-bae3-1dff76312415\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7" Jul 27 16:59:17.364338 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:17.364287 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcfxl\" (UniqueName: \"kubernetes.io/projected/9549a682-fb22-422c-bae3-1dff76312415-kube-api-access-rcfxl\") pod \"payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7\" (UID: \"9549a682-fb22-422c-bae3-1dff76312415\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7" Jul 27 16:59:17.364985 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:17.364960 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9549a682-fb22-422c-bae3-1dff76312415-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7\" (UID: \"9549a682-fb22-422c-bae3-1dff76312415\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7" Jul 27 16:59:17.376428 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:17.376393 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcfxl\" (UniqueName: \"kubernetes.io/projected/9549a682-fb22-422c-bae3-1dff76312415-kube-api-access-rcfxl\") pod \"payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7\" (UID: \"9549a682-fb22-422c-bae3-1dff76312415\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7" Jul 27 16:59:17.427889 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:17.427849 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7" Jul 27 16:59:17.579393 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:17.579363 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7"] Jul 27 16:59:17.581320 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:59:17.581293 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9549a682_fb22_422c_bae3_1dff76312415.slice/crio-088b9ca91330671902a0eab546e7faecc138821349de216954d28a801cf72a0f WatchSource:0}: Error finding container 088b9ca91330671902a0eab546e7faecc138821349de216954d28a801cf72a0f: Status 404 returned error can't find the container with id 088b9ca91330671902a0eab546e7faecc138821349de216954d28a801cf72a0f Jul 27 16:59:18.161952 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:18.161892 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7" event={"ID":"9549a682-fb22-422c-bae3-1dff76312415","Type":"ContainerStarted","Data":"bdce0e07eda7261ecb3f118f581732dcd4bc3959104772872951819f09e31153"} Jul 27 16:59:18.162436 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:18.162073 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7" Jul 27 16:59:18.162436 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:18.162105 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7" event={"ID":"9549a682-fb22-422c-bae3-1dff76312415","Type":"ContainerStarted","Data":"088b9ca91330671902a0eab546e7faecc138821349de216954d28a801cf72a0f"} Jul 27 16:59:18.190427 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:18.190380 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7" podStartSLOduration=1.190364025 podStartE2EDuration="1.190364025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:59:18.190087128 +0000 UTC m=+2536.862621627" watchObservedRunningTime="2026-07-27 16:59:18.190364025 +0000 UTC m=+2536.862898524" Jul 27 16:59:25.692343 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:25.692307 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn"] Jul 27 16:59:25.696572 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:25.696523 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn" Jul 27 16:59:25.700065 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:25.700038 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-d4529a-dockercfg-59ppk\"" Jul 27 16:59:25.700265 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:25.700038 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-d4529a\"" Jul 27 16:59:25.705550 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:25.705523 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn"] Jul 27 16:59:25.851784 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:25.851741 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8fbb\" (UniqueName: \"kubernetes.io/projected/cf24f376-193e-4fab-87a4-f89c13bca1a0-kube-api-access-v8fbb\") pod \"payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn\" (UID: \"cf24f376-193e-4fab-87a4-f89c13bca1a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn" Jul 27 16:59:25.852067 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:25.851931 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf24f376-193e-4fab-87a4-f89c13bca1a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn\" (UID: \"cf24f376-193e-4fab-87a4-f89c13bca1a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn" Jul 27 16:59:25.953690 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:25.953586 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8fbb\" (UniqueName: \"kubernetes.io/projected/cf24f376-193e-4fab-87a4-f89c13bca1a0-kube-api-access-v8fbb\") pod \"payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn\" (UID: \"cf24f376-193e-4fab-87a4-f89c13bca1a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn" Jul 27 16:59:25.953690 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:25.953687 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf24f376-193e-4fab-87a4-f89c13bca1a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn\" (UID: \"cf24f376-193e-4fab-87a4-f89c13bca1a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn" Jul 27 16:59:25.954475 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:25.954450 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf24f376-193e-4fab-87a4-f89c13bca1a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn\" (UID: \"cf24f376-193e-4fab-87a4-f89c13bca1a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn" Jul 27 16:59:25.963522 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:25.963498 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8fbb\" (UniqueName: \"kubernetes.io/projected/cf24f376-193e-4fab-87a4-f89c13bca1a0-kube-api-access-v8fbb\") pod \"payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn\" (UID: \"cf24f376-193e-4fab-87a4-f89c13bca1a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn" Jul 27 16:59:26.010287 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:26.010250 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn" Jul 27 16:59:26.159872 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:26.159837 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn"] Jul 27 16:59:26.161591 ip-10-0-135-70 kubenswrapper[2550]: W0727 16:59:26.161562 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf24f376_193e_4fab_87a4_f89c13bca1a0.slice/crio-7fa7ea3b1fe55021b4d26ef844aa0fc5a6d1f7d0ba4b2c8c968da53ecab37ebb WatchSource:0}: Error finding container 7fa7ea3b1fe55021b4d26ef844aa0fc5a6d1f7d0ba4b2c8c968da53ecab37ebb: Status 404 returned error can't find the container with id 7fa7ea3b1fe55021b4d26ef844aa0fc5a6d1f7d0ba4b2c8c968da53ecab37ebb Jul 27 16:59:26.219414 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:26.219376 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn" event={"ID":"cf24f376-193e-4fab-87a4-f89c13bca1a0","Type":"ContainerStarted","Data":"7fa7ea3b1fe55021b4d26ef844aa0fc5a6d1f7d0ba4b2c8c968da53ecab37ebb"} Jul 27 16:59:27.227550 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:27.227511 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn" event={"ID":"cf24f376-193e-4fab-87a4-f89c13bca1a0","Type":"ContainerStarted","Data":"de01ab09bb94ba361ba7bd519deece9bd8b1b1808ea8ba18a34df90458d12c9b"} Jul 27 16:59:27.228063 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:27.227625 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn" Jul 27 16:59:27.249620 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:27.249568 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn" podStartSLOduration=2.249552944 podStartE2EDuration="2.249552944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:59:27.247019317 +0000 UTC m=+2545.919553816" watchObservedRunningTime="2026-07-27 16:59:27.249552944 +0000 UTC m=+2545.922087442" Jul 27 16:59:29.170055 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:29.170023 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7" Jul 27 16:59:38.237116 ip-10-0-135-70 kubenswrapper[2550]: I0727 16:59:38.237085 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn" Jul 27 17:00:00.139524 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:00.139486 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752860-l2mzl"] Jul 27 17:00:00.143733 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:00.143714 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d4529a-29752860-cbjkh"] Jul 27 17:00:00.143871 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:00.143848 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752860-l2mzl" Jul 27 17:00:00.146955 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:00.146936 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v2ffk\"" Jul 27 17:00:00.147346 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:00.147332 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f1b699-29752860-tvvkx"] Jul 27 17:00:00.147460 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:00.147446 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d4529a-29752860-cbjkh" Jul 27 17:00:00.150938 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:00.150894 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f1b699-29752860-tvvkx" Jul 27 17:00:00.154160 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:00.154129 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752860-l2mzl"] Jul 27 17:00:00.157875 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:00.157844 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f1b699-29752860-tvvkx"] Jul 27 17:00:00.158549 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:00.158527 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752860-l2mzl" Jul 27 17:00:00.162172 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:00.162154 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d4529a-29752860-cbjkh"] Jul 27 17:00:00.169710 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:00.169690 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d4529a-29752860-cbjkh" Jul 27 17:00:00.210884 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:00.210822 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f1b699-29752860-tvvkx" Jul 27 17:00:00.369140 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:00.369061 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d4529a-29752860-cbjkh"] Jul 27 17:00:00.371398 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:00:00.371370 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62432d5_2d94_421a_a505_5c319ff0c8f2.slice/crio-090461eca5febd360df76b4f3d6162eefa76d7fb941df92e90a5832085158ac1 WatchSource:0}: Error finding container 090461eca5febd360df76b4f3d6162eefa76d7fb941df92e90a5832085158ac1: Status 404 returned error can't find the container with id 090461eca5febd360df76b4f3d6162eefa76d7fb941df92e90a5832085158ac1 Jul 27 17:00:00.400059 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:00.399852 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f1b699-29752860-tvvkx"] Jul 27 17:00:00.409238 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:00:00.409212 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce42b508_a489_4ca7_beb8_c67c3a4e18fb.slice/crio-f8ab6e2079bcff0b5e6e31ea8cf3877373dad43da1a6a43f156fea5cc4cc5567 WatchSource:0}: Error finding container f8ab6e2079bcff0b5e6e31ea8cf3877373dad43da1a6a43f156fea5cc4cc5567: Status 404 returned error can't find the container with id f8ab6e2079bcff0b5e6e31ea8cf3877373dad43da1a6a43f156fea5cc4cc5567 Jul 27 17:00:00.444970 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:00.444933 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f1b699-29752860-tvvkx" event={"ID":"ce42b508-a489-4ca7-beb8-c67c3a4e18fb","Type":"ContainerStarted","Data":"f8ab6e2079bcff0b5e6e31ea8cf3877373dad43da1a6a43f156fea5cc4cc5567"} Jul 27 17:00:00.446486 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:00.446461 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d4529a-29752860-cbjkh" event={"ID":"a62432d5-2d94-421a-a505-5c319ff0c8f2","Type":"ContainerStarted","Data":"090461eca5febd360df76b4f3d6162eefa76d7fb941df92e90a5832085158ac1"} Jul 27 17:00:00.542731 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:00.542699 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752860-l2mzl"] Jul 27 17:00:00.543766 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:00:00.543739 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9fd0a8_4ecc_4fc1_b035_0e0bec728aab.slice/crio-a60459147feb0ca4c2100018a1b52737682b5c279a629ed481a13b757d09f809 WatchSource:0}: Error finding container a60459147feb0ca4c2100018a1b52737682b5c279a629ed481a13b757d09f809: Status 404 returned error can't find the container with id a60459147feb0ca4c2100018a1b52737682b5c279a629ed481a13b757d09f809 Jul 27 17:00:01.454561 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:01.454522 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752860-l2mzl" event={"ID":"2b9fd0a8-4ecc-4fc1-b035-0e0bec728aab","Type":"ContainerStarted","Data":"6c7003c4fe44873b50d187ab4bc63d692cd4332578b7552958d3f9579f67b0d5"} Jul 27 17:00:01.454998 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:01.454581 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752860-l2mzl" event={"ID":"2b9fd0a8-4ecc-4fc1-b035-0e0bec728aab","Type":"ContainerStarted","Data":"a60459147feb0ca4c2100018a1b52737682b5c279a629ed481a13b757d09f809"} Jul 27 17:00:01.456172 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:01.456142 2550 generic.go:356] "Generic (PLEG): container finished" podID="ce42b508-a489-4ca7-beb8-c67c3a4e18fb" containerID="88c73010b9df3af0e83a2f2e2f1db9e1168ffb1b2e1e9e90bed9f35d1983dd7a" exitCode=0 Jul 27 17:00:01.456288 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:01.456217 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f1b699-29752860-tvvkx" event={"ID":"ce42b508-a489-4ca7-beb8-c67c3a4e18fb","Type":"ContainerDied","Data":"88c73010b9df3af0e83a2f2e2f1db9e1168ffb1b2e1e9e90bed9f35d1983dd7a"} Jul 27 17:00:01.457703 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:01.457683 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d4529a-29752860-cbjkh" event={"ID":"a62432d5-2d94-421a-a505-5c319ff0c8f2","Type":"ContainerStarted","Data":"50a8a826f8cec5aca449b3dd31fd4af43b9ada022d5b8bccca63b483270f1fc7"} Jul 27 17:00:01.471968 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:01.471923 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752860-l2mzl" podStartSLOduration=1.471891134 podStartE2EDuration="1.471891134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:00:01.469830383 +0000 UTC m=+2580.142364884" watchObservedRunningTime="2026-07-27 17:00:01.471891134 +0000 UTC m=+2580.144425632" Jul 27 17:00:01.490459 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:01.490414 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d4529a-29752860-cbjkh" podStartSLOduration=1.490401007 podStartE2EDuration="1.490401007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:00:01.490125 +0000 UTC m=+2580.162659500" watchObservedRunningTime="2026-07-27 17:00:01.490401007 +0000 UTC m=+2580.162935506" Jul 27 17:00:02.285459 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:02.285419 2550 scope.go:117] "RemoveContainer" containerID="f1b23423fa81453b304b1410c1a796cb8807088d1ccdd45c37757d55e6d4c320" Jul 27 17:00:02.466063 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:02.466022 2550 generic.go:356] "Generic (PLEG): container finished" podID="2b9fd0a8-4ecc-4fc1-b035-0e0bec728aab" containerID="6c7003c4fe44873b50d187ab4bc63d692cd4332578b7552958d3f9579f67b0d5" exitCode=0 Jul 27 17:00:02.466529 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:02.466102 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752860-l2mzl" event={"ID":"2b9fd0a8-4ecc-4fc1-b035-0e0bec728aab","Type":"ContainerDied","Data":"6c7003c4fe44873b50d187ab4bc63d692cd4332578b7552958d3f9579f67b0d5"} Jul 27 17:00:02.467671 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:02.467647 2550 generic.go:356] "Generic (PLEG): container finished" podID="a62432d5-2d94-421a-a505-5c319ff0c8f2" containerID="50a8a826f8cec5aca449b3dd31fd4af43b9ada022d5b8bccca63b483270f1fc7" exitCode=0 Jul 27 17:00:02.467757 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:02.467729 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d4529a-29752860-cbjkh" event={"ID":"a62432d5-2d94-421a-a505-5c319ff0c8f2","Type":"ContainerDied","Data":"50a8a826f8cec5aca449b3dd31fd4af43b9ada022d5b8bccca63b483270f1fc7"} Jul 27 17:00:02.613967 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:02.613945 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f1b699-29752860-tvvkx" Jul 27 17:00:03.475709 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:03.475674 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f1b699-29752860-tvvkx" Jul 27 17:00:03.476158 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:03.475712 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f1b699-29752860-tvvkx" event={"ID":"ce42b508-a489-4ca7-beb8-c67c3a4e18fb","Type":"ContainerDied","Data":"f8ab6e2079bcff0b5e6e31ea8cf3877373dad43da1a6a43f156fea5cc4cc5567"} Jul 27 17:00:03.476158 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:03.475763 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8ab6e2079bcff0b5e6e31ea8cf3877373dad43da1a6a43f156fea5cc4cc5567" Jul 27 17:00:03.625520 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:03.625498 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d4529a-29752860-cbjkh" Jul 27 17:00:03.628886 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:03.628866 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752860-l2mzl" Jul 27 17:00:04.483101 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:04.483073 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752860-l2mzl" Jul 27 17:00:04.483544 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:04.483118 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752860-l2mzl" event={"ID":"2b9fd0a8-4ecc-4fc1-b035-0e0bec728aab","Type":"ContainerDied","Data":"a60459147feb0ca4c2100018a1b52737682b5c279a629ed481a13b757d09f809"} Jul 27 17:00:04.483544 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:04.483175 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a60459147feb0ca4c2100018a1b52737682b5c279a629ed481a13b757d09f809" Jul 27 17:00:04.484536 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:04.484504 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d4529a-29752860-cbjkh" event={"ID":"a62432d5-2d94-421a-a505-5c319ff0c8f2","Type":"ContainerDied","Data":"090461eca5febd360df76b4f3d6162eefa76d7fb941df92e90a5832085158ac1"} Jul 27 17:00:04.484536 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:04.484532 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d4529a-29752860-cbjkh" Jul 27 17:00:04.484662 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:00:04.484554 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="090461eca5febd360df76b4f3d6162eefa76d7fb941df92e90a5832085158ac1" Jul 27 17:02:02.064732 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:02:02.064704 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 17:02:02.074458 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:02:02.074435 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 17:04:02.339662 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:04:02.339627 2550 scope.go:117] "RemoveContainer" containerID="d6149ab3d014478919bf8a591b556b1adeb97f87494fdb4b9e33cb0c0e881192" Jul 27 17:04:02.349381 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:04:02.349363 2550 scope.go:117] "RemoveContainer" containerID="fc76e607f49cb14f8ed232f4679768ce5c6d496a20687c29dc49d18062f69508" Jul 27 17:05:02.377255 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:05:02.377218 2550 scope.go:117] "RemoveContainer" containerID="d6438b2dfaaebddbd459d413ea0f1eca8a87f36a389a4a8a13b328be0a32347e" Jul 27 17:05:02.386784 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:05:02.386758 2550 scope.go:117] "RemoveContainer" containerID="aef27bbd534c54900488ac977ea3fa4677412b9a7559613e8f9568677dfae1e8" Jul 27 17:07:02.108763 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:07:02.108730 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 17:07:02.118493 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:07:02.118472 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 17:08:06.628632 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:06.628595 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7564-139639731745712"] Jul 27 17:08:06.629118 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:06.629061 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b9fd0a8-4ecc-4fc1-b035-0e0bec728aab" containerName="cleanup" Jul 27 17:08:06.629118 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:06.629073 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b9fd0a8-4ecc-4fc1-b035-0e0bec728aab" containerName="cleanup" Jul 27 17:08:06.629118 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:06.629095 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62432d5-2d94-421a-a505-5c319ff0c8f2" containerName="cleanup" Jul 27 17:08:06.629118 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:06.629100 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62432d5-2d94-421a-a505-5c319ff0c8f2" containerName="cleanup" Jul 27 17:08:06.629118 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:06.629110 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce42b508-a489-4ca7-beb8-c67c3a4e18fb" containerName="cleanup" Jul 27 17:08:06.629118 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:06.629114 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce42b508-a489-4ca7-beb8-c67c3a4e18fb" containerName="cleanup" Jul 27 17:08:06.629311 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:06.629200 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b9fd0a8-4ecc-4fc1-b035-0e0bec728aab" containerName="cleanup" Jul 27 17:08:06.629311 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:06.629209 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce42b508-a489-4ca7-beb8-c67c3a4e18fb" containerName="cleanup" Jul 27 17:08:06.629311 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:06.629215 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a62432d5-2d94-421a-a505-5c319ff0c8f2" containerName="cleanup" Jul 27 17:08:06.632538 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:06.632521 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731745712" Jul 27 17:08:06.635509 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:06.635488 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-l7m5q\"" Jul 27 17:08:06.638678 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:06.638652 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-139639731745712"] Jul 27 17:08:06.682956 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:06.682891 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzxbm\" (UniqueName: \"kubernetes.io/projected/c0c06593-410a-41e9-86a7-1cc7cd396ec7-kube-api-access-tzxbm\") pod \"test-curl-7564-139639731745712\" (UID: \"c0c06593-410a-41e9-86a7-1cc7cd396ec7\") " pod="opendatahub/test-curl-7564-139639731745712" Jul 27 17:08:06.784246 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:06.784200 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzxbm\" (UniqueName: \"kubernetes.io/projected/c0c06593-410a-41e9-86a7-1cc7cd396ec7-kube-api-access-tzxbm\") pod \"test-curl-7564-139639731745712\" (UID: \"c0c06593-410a-41e9-86a7-1cc7cd396ec7\") " pod="opendatahub/test-curl-7564-139639731745712" Jul 27 17:08:06.795101 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:06.795063 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzxbm\" (UniqueName: \"kubernetes.io/projected/c0c06593-410a-41e9-86a7-1cc7cd396ec7-kube-api-access-tzxbm\") pod \"test-curl-7564-139639731745712\" (UID: \"c0c06593-410a-41e9-86a7-1cc7cd396ec7\") " pod="opendatahub/test-curl-7564-139639731745712" Jul 27 17:08:06.944240 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:06.944151 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731745712" Jul 27 17:08:07.092044 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:07.092015 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-139639731745712"] Jul 27 17:08:07.093081 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:08:07.093052 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0c06593_410a_41e9_86a7_1cc7cd396ec7.slice/crio-c18649c5d02ffe66ca9707b58b967c598faba1add43b5e548b9cb25f9280ec3f WatchSource:0}: Error finding container c18649c5d02ffe66ca9707b58b967c598faba1add43b5e548b9cb25f9280ec3f: Status 404 returned error can't find the container with id c18649c5d02ffe66ca9707b58b967c598faba1add43b5e548b9cb25f9280ec3f Jul 27 17:08:07.094476 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:07.094456 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 17:08:07.583284 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:07.583234 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-139639731745712" event={"ID":"c0c06593-410a-41e9-86a7-1cc7cd396ec7","Type":"ContainerStarted","Data":"c18649c5d02ffe66ca9707b58b967c598faba1add43b5e548b9cb25f9280ec3f"} Jul 27 17:08:08.590804 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:08.590763 2550 generic.go:356] "Generic (PLEG): container finished" podID="c0c06593-410a-41e9-86a7-1cc7cd396ec7" containerID="df5bd4c3d5f0371d305e448431ca8dbf7dd970b0ac58b5b69967e7ee1ff63a2a" exitCode=0 Jul 27 17:08:08.591226 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:08.590848 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-139639731745712" event={"ID":"c0c06593-410a-41e9-86a7-1cc7cd396ec7","Type":"ContainerDied","Data":"df5bd4c3d5f0371d305e448431ca8dbf7dd970b0ac58b5b69967e7ee1ff63a2a"} Jul 27 17:08:09.742806 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:09.742773 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731745712" Jul 27 17:08:09.817656 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:09.817625 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzxbm\" (UniqueName: \"kubernetes.io/projected/c0c06593-410a-41e9-86a7-1cc7cd396ec7-kube-api-access-tzxbm\") pod \"c0c06593-410a-41e9-86a7-1cc7cd396ec7\" (UID: \"c0c06593-410a-41e9-86a7-1cc7cd396ec7\") " Jul 27 17:08:09.820054 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:09.820027 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c06593-410a-41e9-86a7-1cc7cd396ec7-kube-api-access-tzxbm" (OuterVolumeSpecName: "kube-api-access-tzxbm") pod "c0c06593-410a-41e9-86a7-1cc7cd396ec7" (UID: "c0c06593-410a-41e9-86a7-1cc7cd396ec7"). InnerVolumeSpecName "kube-api-access-tzxbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:08:09.820648 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:09.820624 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7564-139639731745712_c0c06593-410a-41e9-86a7-1cc7cd396ec7/test-curl-7564-139639731745712/0.log" Jul 27 17:08:09.861334 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:09.861304 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7564-139639731745712"] Jul 27 17:08:09.863897 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:09.863876 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7564-139639731745712"] Jul 27 17:08:09.919537 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:09.919506 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzxbm\" (UniqueName: \"kubernetes.io/projected/c0c06593-410a-41e9-86a7-1cc7cd396ec7-kube-api-access-tzxbm\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:08:09.991787 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:09.991754 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7564-139639731830832"] Jul 27 17:08:09.992283 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:09.992268 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0c06593-410a-41e9-86a7-1cc7cd396ec7" containerName="test-curl-7564-139639731745712" Jul 27 17:08:09.992331 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:09.992285 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0c06593-410a-41e9-86a7-1cc7cd396ec7" containerName="test-curl-7564-139639731745712" Jul 27 17:08:09.992375 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:09.992362 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0c06593-410a-41e9-86a7-1cc7cd396ec7" containerName="test-curl-7564-139639731745712" Jul 27 17:08:09.995654 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:09.995589 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731830832" Jul 27 17:08:10.000609 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:10.000583 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-139639731830832"] Jul 27 17:08:10.020734 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:10.020698 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb6jj\" (UniqueName: \"kubernetes.io/projected/e7c0775c-04d6-4ba8-b353-c30948c18f50-kube-api-access-zb6jj\") pod \"test-curl-7564-139639731830832\" (UID: \"e7c0775c-04d6-4ba8-b353-c30948c18f50\") " pod="opendatahub/test-curl-7564-139639731830832" Jul 27 17:08:10.121405 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:10.121323 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb6jj\" (UniqueName: \"kubernetes.io/projected/e7c0775c-04d6-4ba8-b353-c30948c18f50-kube-api-access-zb6jj\") pod \"test-curl-7564-139639731830832\" (UID: \"e7c0775c-04d6-4ba8-b353-c30948c18f50\") " pod="opendatahub/test-curl-7564-139639731830832" Jul 27 17:08:10.136300 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:10.136272 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb6jj\" (UniqueName: \"kubernetes.io/projected/e7c0775c-04d6-4ba8-b353-c30948c18f50-kube-api-access-zb6jj\") pod \"test-curl-7564-139639731830832\" (UID: \"e7c0775c-04d6-4ba8-b353-c30948c18f50\") " pod="opendatahub/test-curl-7564-139639731830832" Jul 27 17:08:10.308163 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:10.308120 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731830832" Jul 27 17:08:10.455197 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:10.455174 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-139639731830832"] Jul 27 17:08:10.456568 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:08:10.456542 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7c0775c_04d6_4ba8_b353_c30948c18f50.slice/crio-18e91ff66b9d341d0cee4fa1a9bd00a11e53b0b59276bf79ff104d62f4f16314 WatchSource:0}: Error finding container 18e91ff66b9d341d0cee4fa1a9bd00a11e53b0b59276bf79ff104d62f4f16314: Status 404 returned error can't find the container with id 18e91ff66b9d341d0cee4fa1a9bd00a11e53b0b59276bf79ff104d62f4f16314 Jul 27 17:08:10.607377 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:10.607350 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c18649c5d02ffe66ca9707b58b967c598faba1add43b5e548b9cb25f9280ec3f" Jul 27 17:08:10.607377 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:10.607372 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731745712" Jul 27 17:08:10.608605 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:10.608580 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-139639731830832" event={"ID":"e7c0775c-04d6-4ba8-b353-c30948c18f50","Type":"ContainerStarted","Data":"18e91ff66b9d341d0cee4fa1a9bd00a11e53b0b59276bf79ff104d62f4f16314"} Jul 27 17:08:10.613191 ip-10-0-135-70 kubenswrapper[2550]: E0727 17:08:10.613164 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-139639731745712\" is forbidden: User \"system:node:ip-10-0-135-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-70.ec2.internal' and this object" podUID="c0c06593-410a-41e9-86a7-1cc7cd396ec7" pod="opendatahub/test-curl-7564-139639731745712" Jul 27 17:08:11.617505 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:11.617470 2550 generic.go:356] "Generic (PLEG): container finished" podID="e7c0775c-04d6-4ba8-b353-c30948c18f50" containerID="b08e209df07c302a452e2895597bad268fc31f91d234935a223f263860c46331" exitCode=0 Jul 27 17:08:11.617890 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:11.617563 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-139639731830832" event={"ID":"e7c0775c-04d6-4ba8-b353-c30948c18f50","Type":"ContainerDied","Data":"b08e209df07c302a452e2895597bad268fc31f91d234935a223f263860c46331"} Jul 27 17:08:11.643971 ip-10-0-135-70 kubenswrapper[2550]: E0727 17:08:11.643929 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-139639731745712\" is forbidden: User \"system:node:ip-10-0-135-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-70.ec2.internal' and this object" podUID="c0c06593-410a-41e9-86a7-1cc7cd396ec7" pod="opendatahub/test-curl-7564-139639731745712" Jul 27 17:08:11.856927 ip-10-0-135-70 kubenswrapper[2550]: E0727 17:08:11.856867 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-139639731745712\" is forbidden: User \"system:node:ip-10-0-135-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-70.ec2.internal' and this object" podUID="c0c06593-410a-41e9-86a7-1cc7cd396ec7" pod="opendatahub/test-curl-7564-139639731745712" Jul 27 17:08:11.856927 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:11.856869 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0c06593-410a-41e9-86a7-1cc7cd396ec7" path="/var/lib/kubelet/pods/c0c06593-410a-41e9-86a7-1cc7cd396ec7/volumes" Jul 27 17:08:12.777124 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:12.777090 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731830832" Jul 27 17:08:12.824736 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:12.824701 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7564-139639731830832_e7c0775c-04d6-4ba8-b353-c30948c18f50/test-curl-7564-139639731830832/0.log" Jul 27 17:08:12.849506 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:12.849479 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb6jj\" (UniqueName: \"kubernetes.io/projected/e7c0775c-04d6-4ba8-b353-c30948c18f50-kube-api-access-zb6jj\") pod \"e7c0775c-04d6-4ba8-b353-c30948c18f50\" (UID: \"e7c0775c-04d6-4ba8-b353-c30948c18f50\") " Jul 27 17:08:12.851693 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:12.851670 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c0775c-04d6-4ba8-b353-c30948c18f50-kube-api-access-zb6jj" (OuterVolumeSpecName: "kube-api-access-zb6jj") pod "e7c0775c-04d6-4ba8-b353-c30948c18f50" (UID: "e7c0775c-04d6-4ba8-b353-c30948c18f50"). InnerVolumeSpecName "kube-api-access-zb6jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:08:12.855889 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:12.855860 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7564-139639731830832"] Jul 27 17:08:12.861338 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:12.861317 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7564-139639731830832"] Jul 27 17:08:12.951221 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:12.951143 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb6jj\" (UniqueName: \"kubernetes.io/projected/e7c0775c-04d6-4ba8-b353-c30948c18f50-kube-api-access-zb6jj\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:08:13.061876 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:13.061843 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7564-139639731873664"] Jul 27 17:08:13.062427 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:13.062408 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c0775c-04d6-4ba8-b353-c30948c18f50" containerName="test-curl-7564-139639731830832" Jul 27 17:08:13.062528 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:13.062429 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c0775c-04d6-4ba8-b353-c30948c18f50" containerName="test-curl-7564-139639731830832" Jul 27 17:08:13.062585 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:13.062541 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c0775c-04d6-4ba8-b353-c30948c18f50" containerName="test-curl-7564-139639731830832" Jul 27 17:08:13.065796 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:13.065776 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731873664" Jul 27 17:08:13.126279 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:13.126246 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-139639731873664"] Jul 27 17:08:13.153125 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:13.153086 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sj4r\" (UniqueName: \"kubernetes.io/projected/3c096b27-b3d7-47b6-92c8-025ef74b01e4-kube-api-access-6sj4r\") pod \"test-curl-7564-139639731873664\" (UID: \"3c096b27-b3d7-47b6-92c8-025ef74b01e4\") " pod="opendatahub/test-curl-7564-139639731873664" Jul 27 17:08:13.254316 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:13.254235 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sj4r\" (UniqueName: \"kubernetes.io/projected/3c096b27-b3d7-47b6-92c8-025ef74b01e4-kube-api-access-6sj4r\") pod \"test-curl-7564-139639731873664\" (UID: \"3c096b27-b3d7-47b6-92c8-025ef74b01e4\") " pod="opendatahub/test-curl-7564-139639731873664" Jul 27 17:08:13.266771 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:13.266745 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sj4r\" (UniqueName: \"kubernetes.io/projected/3c096b27-b3d7-47b6-92c8-025ef74b01e4-kube-api-access-6sj4r\") pod \"test-curl-7564-139639731873664\" (UID: \"3c096b27-b3d7-47b6-92c8-025ef74b01e4\") " pod="opendatahub/test-curl-7564-139639731873664" Jul 27 17:08:13.375809 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:13.375776 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731873664" Jul 27 17:08:13.528639 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:13.528606 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-139639731873664"] Jul 27 17:08:13.530075 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:08:13.530052 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c096b27_b3d7_47b6_92c8_025ef74b01e4.slice/crio-c8e2f672549014656858c2a0c6e9fc92b7565ec69443db9cc41e3edfcfafa630 WatchSource:0}: Error finding container c8e2f672549014656858c2a0c6e9fc92b7565ec69443db9cc41e3edfcfafa630: Status 404 returned error can't find the container with id c8e2f672549014656858c2a0c6e9fc92b7565ec69443db9cc41e3edfcfafa630 Jul 27 17:08:13.631842 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:13.631809 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731830832" Jul 27 17:08:13.631842 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:13.631823 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18e91ff66b9d341d0cee4fa1a9bd00a11e53b0b59276bf79ff104d62f4f16314" Jul 27 17:08:13.633323 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:13.633289 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-139639731873664" event={"ID":"3c096b27-b3d7-47b6-92c8-025ef74b01e4","Type":"ContainerStarted","Data":"c8e2f672549014656858c2a0c6e9fc92b7565ec69443db9cc41e3edfcfafa630"} Jul 27 17:08:13.643144 ip-10-0-135-70 kubenswrapper[2550]: E0727 17:08:13.643119 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-139639731830832\" is forbidden: User \"system:node:ip-10-0-135-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-70.ec2.internal' and this object" podUID="e7c0775c-04d6-4ba8-b353-c30948c18f50" pod="opendatahub/test-curl-7564-139639731830832" Jul 27 17:08:13.851078 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:13.851048 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7c0775c-04d6-4ba8-b353-c30948c18f50" path="/var/lib/kubelet/pods/e7c0775c-04d6-4ba8-b353-c30948c18f50/volumes" Jul 27 17:08:14.641641 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:14.641603 2550 generic.go:356] "Generic (PLEG): container finished" podID="3c096b27-b3d7-47b6-92c8-025ef74b01e4" containerID="7a929fcbf36c0d709ea224fb49e76385c1523d4c334fced281f83a86dc62ad5c" exitCode=0 Jul 27 17:08:14.641823 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:14.641696 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-139639731873664" event={"ID":"3c096b27-b3d7-47b6-92c8-025ef74b01e4","Type":"ContainerDied","Data":"7a929fcbf36c0d709ea224fb49e76385c1523d4c334fced281f83a86dc62ad5c"} Jul 27 17:08:15.796701 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:15.796673 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731873664" Jul 27 17:08:15.820851 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:15.820826 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7564-139639731873664_3c096b27-b3d7-47b6-92c8-025ef74b01e4/test-curl-7564-139639731873664/0.log" Jul 27 17:08:15.854770 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:15.854742 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7564-139639731873664"] Jul 27 17:08:15.854949 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:15.854828 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7564-139639731873664"] Jul 27 17:08:15.878141 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:15.878109 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sj4r\" (UniqueName: \"kubernetes.io/projected/3c096b27-b3d7-47b6-92c8-025ef74b01e4-kube-api-access-6sj4r\") pod \"3c096b27-b3d7-47b6-92c8-025ef74b01e4\" (UID: \"3c096b27-b3d7-47b6-92c8-025ef74b01e4\") " Jul 27 17:08:15.880255 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:15.880225 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c096b27-b3d7-47b6-92c8-025ef74b01e4-kube-api-access-6sj4r" (OuterVolumeSpecName: "kube-api-access-6sj4r") pod "3c096b27-b3d7-47b6-92c8-025ef74b01e4" (UID: "3c096b27-b3d7-47b6-92c8-025ef74b01e4"). InnerVolumeSpecName "kube-api-access-6sj4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:08:15.977558 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:15.977476 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7564-139639731873664"] Jul 27 17:08:15.977969 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:15.977953 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c096b27-b3d7-47b6-92c8-025ef74b01e4" containerName="test-curl-7564-139639731873664" Jul 27 17:08:15.978019 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:15.977970 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c096b27-b3d7-47b6-92c8-025ef74b01e4" containerName="test-curl-7564-139639731873664" Jul 27 17:08:15.978075 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:15.978066 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c096b27-b3d7-47b6-92c8-025ef74b01e4" containerName="test-curl-7564-139639731873664" Jul 27 17:08:15.979830 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:15.979808 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sj4r\" (UniqueName: \"kubernetes.io/projected/3c096b27-b3d7-47b6-92c8-025ef74b01e4-kube-api-access-6sj4r\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:08:15.981193 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:15.981179 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731873664" Jul 27 17:08:15.988504 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:15.988476 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-139639731873664"] Jul 27 17:08:16.081155 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:16.081115 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq789\" (UniqueName: \"kubernetes.io/projected/094a0e33-5ceb-4822-88c8-f3630e1d268f-kube-api-access-wq789\") pod \"test-curl-7564-139639731873664\" (UID: \"094a0e33-5ceb-4822-88c8-f3630e1d268f\") " pod="opendatahub/test-curl-7564-139639731873664" Jul 27 17:08:16.182597 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:16.182565 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq789\" (UniqueName: \"kubernetes.io/projected/094a0e33-5ceb-4822-88c8-f3630e1d268f-kube-api-access-wq789\") pod \"test-curl-7564-139639731873664\" (UID: \"094a0e33-5ceb-4822-88c8-f3630e1d268f\") " pod="opendatahub/test-curl-7564-139639731873664" Jul 27 17:08:16.192453 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:16.192424 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq789\" (UniqueName: \"kubernetes.io/projected/094a0e33-5ceb-4822-88c8-f3630e1d268f-kube-api-access-wq789\") pod \"test-curl-7564-139639731873664\" (UID: \"094a0e33-5ceb-4822-88c8-f3630e1d268f\") " pod="opendatahub/test-curl-7564-139639731873664" Jul 27 17:08:16.292160 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:16.292127 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731873664" Jul 27 17:08:16.437163 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:16.437135 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-139639731873664"] Jul 27 17:08:16.438349 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:08:16.438322 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod094a0e33_5ceb_4822_88c8_f3630e1d268f.slice/crio-f6df5cb021bf7226302ec0cb93c0f4159eaa6ca39f840d1d96d9878980332672 WatchSource:0}: Error finding container f6df5cb021bf7226302ec0cb93c0f4159eaa6ca39f840d1d96d9878980332672: Status 404 returned error can't find the container with id f6df5cb021bf7226302ec0cb93c0f4159eaa6ca39f840d1d96d9878980332672 Jul 27 17:08:16.655017 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:16.654932 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-139639731873664" event={"ID":"094a0e33-5ceb-4822-88c8-f3630e1d268f","Type":"ContainerStarted","Data":"f6df5cb021bf7226302ec0cb93c0f4159eaa6ca39f840d1d96d9878980332672"} Jul 27 17:08:16.656168 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:16.656149 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8e2f672549014656858c2a0c6e9fc92b7565ec69443db9cc41e3edfcfafa630" Jul 27 17:08:16.656288 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:16.656204 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731873664" Jul 27 17:08:16.668351 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:16.668320 2550 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7564-139639731873664" oldPodUID="3c096b27-b3d7-47b6-92c8-025ef74b01e4" podUID="094a0e33-5ceb-4822-88c8-f3630e1d268f" Jul 27 17:08:17.665172 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:17.665135 2550 generic.go:356] "Generic (PLEG): container finished" podID="094a0e33-5ceb-4822-88c8-f3630e1d268f" containerID="6382a387a2e584de14116f5b1d0874448a876221bea9613249038d7524db541c" exitCode=0 Jul 27 17:08:17.665561 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:17.665209 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-139639731873664" event={"ID":"094a0e33-5ceb-4822-88c8-f3630e1d268f","Type":"ContainerDied","Data":"6382a387a2e584de14116f5b1d0874448a876221bea9613249038d7524db541c"} Jul 27 17:08:17.850531 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:17.850501 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c096b27-b3d7-47b6-92c8-025ef74b01e4" path="/var/lib/kubelet/pods/3c096b27-b3d7-47b6-92c8-025ef74b01e4/volumes" Jul 27 17:08:18.825783 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:18.825751 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731873664" Jul 27 17:08:18.851233 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:18.851207 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7564-139639731873664_094a0e33-5ceb-4822-88c8-f3630e1d268f/test-curl-7564-139639731873664/0.log" Jul 27 17:08:18.880324 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:18.880291 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7564-139639731873664"] Jul 27 17:08:18.882955 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:18.882934 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7564-139639731873664"] Jul 27 17:08:18.911337 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:18.911309 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq789\" (UniqueName: \"kubernetes.io/projected/094a0e33-5ceb-4822-88c8-f3630e1d268f-kube-api-access-wq789\") pod \"094a0e33-5ceb-4822-88c8-f3630e1d268f\" (UID: \"094a0e33-5ceb-4822-88c8-f3630e1d268f\") " Jul 27 17:08:18.913466 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:18.913439 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094a0e33-5ceb-4822-88c8-f3630e1d268f-kube-api-access-wq789" (OuterVolumeSpecName: "kube-api-access-wq789") pod "094a0e33-5ceb-4822-88c8-f3630e1d268f" (UID: "094a0e33-5ceb-4822-88c8-f3630e1d268f"). InnerVolumeSpecName "kube-api-access-wq789". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:08:19.005062 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:19.004979 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7564-139639731676768"] Jul 27 17:08:19.005487 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:19.005468 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="094a0e33-5ceb-4822-88c8-f3630e1d268f" containerName="test-curl-7564-139639731873664" Jul 27 17:08:19.005487 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:19.005488 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="094a0e33-5ceb-4822-88c8-f3630e1d268f" containerName="test-curl-7564-139639731873664" Jul 27 17:08:19.005593 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:19.005583 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="094a0e33-5ceb-4822-88c8-f3630e1d268f" containerName="test-curl-7564-139639731873664" Jul 27 17:08:19.008890 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:19.008844 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731676768" Jul 27 17:08:19.012953 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:19.012925 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq789\" (UniqueName: \"kubernetes.io/projected/094a0e33-5ceb-4822-88c8-f3630e1d268f-kube-api-access-wq789\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:08:19.013157 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:19.013138 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-139639731676768"] Jul 27 17:08:19.113933 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:19.113864 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8bv7\" (UniqueName: \"kubernetes.io/projected/e8eb7a92-9a09-4f8b-8a58-1a926676befb-kube-api-access-k8bv7\") pod \"test-curl-7564-139639731676768\" (UID: \"e8eb7a92-9a09-4f8b-8a58-1a926676befb\") " pod="opendatahub/test-curl-7564-139639731676768" Jul 27 17:08:19.214970 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:19.214935 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bv7\" (UniqueName: \"kubernetes.io/projected/e8eb7a92-9a09-4f8b-8a58-1a926676befb-kube-api-access-k8bv7\") pod \"test-curl-7564-139639731676768\" (UID: \"e8eb7a92-9a09-4f8b-8a58-1a926676befb\") " pod="opendatahub/test-curl-7564-139639731676768" Jul 27 17:08:19.225712 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:19.225676 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8bv7\" (UniqueName: \"kubernetes.io/projected/e8eb7a92-9a09-4f8b-8a58-1a926676befb-kube-api-access-k8bv7\") pod \"test-curl-7564-139639731676768\" (UID: \"e8eb7a92-9a09-4f8b-8a58-1a926676befb\") " pod="opendatahub/test-curl-7564-139639731676768" Jul 27 17:08:19.320726 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:19.320689 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731676768" Jul 27 17:08:19.470060 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:19.470027 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-139639731676768"] Jul 27 17:08:19.473169 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:08:19.473124 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8eb7a92_9a09_4f8b_8a58_1a926676befb.slice/crio-5d1a5f329406683c3554c054c624ec289887fae7ba9fed9cac95813d180ca5ef WatchSource:0}: Error finding container 5d1a5f329406683c3554c054c624ec289887fae7ba9fed9cac95813d180ca5ef: Status 404 returned error can't find the container with id 5d1a5f329406683c3554c054c624ec289887fae7ba9fed9cac95813d180ca5ef Jul 27 17:08:19.679331 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:19.679247 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6df5cb021bf7226302ec0cb93c0f4159eaa6ca39f840d1d96d9878980332672" Jul 27 17:08:19.679331 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:19.679247 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731873664" Jul 27 17:08:19.680675 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:19.680647 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-139639731676768" event={"ID":"e8eb7a92-9a09-4f8b-8a58-1a926676befb","Type":"ContainerStarted","Data":"5d1a5f329406683c3554c054c624ec289887fae7ba9fed9cac95813d180ca5ef"} Jul 27 17:08:19.690291 ip-10-0-135-70 kubenswrapper[2550]: E0727 17:08:19.690266 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-139639731873664\" is forbidden: User \"system:node:ip-10-0-135-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-70.ec2.internal' and this object" podUID="094a0e33-5ceb-4822-88c8-f3630e1d268f" pod="opendatahub/test-curl-7564-139639731873664" Jul 27 17:08:19.851381 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:19.851347 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="094a0e33-5ceb-4822-88c8-f3630e1d268f" path="/var/lib/kubelet/pods/094a0e33-5ceb-4822-88c8-f3630e1d268f/volumes" Jul 27 17:08:20.688034 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:20.687991 2550 generic.go:356] "Generic (PLEG): container finished" podID="e8eb7a92-9a09-4f8b-8a58-1a926676befb" containerID="15b8cdb36e8d6ce293d8be579b266e6565a611d911048a712e5719c967c5f8e1" exitCode=0 Jul 27 17:08:20.688219 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:20.688115 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-139639731676768" event={"ID":"e8eb7a92-9a09-4f8b-8a58-1a926676befb","Type":"ContainerDied","Data":"15b8cdb36e8d6ce293d8be579b266e6565a611d911048a712e5719c967c5f8e1"} Jul 27 17:08:21.851645 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:21.851609 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731676768" Jul 27 17:08:21.873283 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:21.873252 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7564-139639731676768_e8eb7a92-9a09-4f8b-8a58-1a926676befb/test-curl-7564-139639731676768/0.log" Jul 27 17:08:21.897955 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:21.897927 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7564-139639731676768"] Jul 27 17:08:21.900724 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:21.900701 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7564-139639731676768"] Jul 27 17:08:21.944979 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:21.944934 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8bv7\" (UniqueName: \"kubernetes.io/projected/e8eb7a92-9a09-4f8b-8a58-1a926676befb-kube-api-access-k8bv7\") pod \"e8eb7a92-9a09-4f8b-8a58-1a926676befb\" (UID: \"e8eb7a92-9a09-4f8b-8a58-1a926676befb\") " Jul 27 17:08:21.947120 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:21.947069 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8eb7a92-9a09-4f8b-8a58-1a926676befb-kube-api-access-k8bv7" (OuterVolumeSpecName: "kube-api-access-k8bv7") pod "e8eb7a92-9a09-4f8b-8a58-1a926676befb" (UID: "e8eb7a92-9a09-4f8b-8a58-1a926676befb"). InnerVolumeSpecName "kube-api-access-k8bv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:08:22.023002 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:22.022969 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7564-139639731676768"] Jul 27 17:08:22.023670 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:22.023646 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8eb7a92-9a09-4f8b-8a58-1a926676befb" containerName="test-curl-7564-139639731676768" Jul 27 17:08:22.023670 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:22.023671 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8eb7a92-9a09-4f8b-8a58-1a926676befb" containerName="test-curl-7564-139639731676768" Jul 27 17:08:22.023828 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:22.023791 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8eb7a92-9a09-4f8b-8a58-1a926676befb" containerName="test-curl-7564-139639731676768" Jul 27 17:08:22.027054 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:22.027035 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731676768" Jul 27 17:08:22.033777 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:22.033750 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-139639731676768"] Jul 27 17:08:22.046782 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:22.046752 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8bv7\" (UniqueName: \"kubernetes.io/projected/e8eb7a92-9a09-4f8b-8a58-1a926676befb-kube-api-access-k8bv7\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:08:22.147770 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:22.147731 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5f9m\" (UniqueName: \"kubernetes.io/projected/a1c8e34c-0d81-4c39-8fd9-886eb313ad20-kube-api-access-x5f9m\") pod \"test-curl-7564-139639731676768\" (UID: \"a1c8e34c-0d81-4c39-8fd9-886eb313ad20\") " pod="opendatahub/test-curl-7564-139639731676768" Jul 27 17:08:22.248763 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:22.248674 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5f9m\" (UniqueName: \"kubernetes.io/projected/a1c8e34c-0d81-4c39-8fd9-886eb313ad20-kube-api-access-x5f9m\") pod \"test-curl-7564-139639731676768\" (UID: \"a1c8e34c-0d81-4c39-8fd9-886eb313ad20\") " pod="opendatahub/test-curl-7564-139639731676768" Jul 27 17:08:22.259159 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:22.259131 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5f9m\" (UniqueName: \"kubernetes.io/projected/a1c8e34c-0d81-4c39-8fd9-886eb313ad20-kube-api-access-x5f9m\") pod \"test-curl-7564-139639731676768\" (UID: \"a1c8e34c-0d81-4c39-8fd9-886eb313ad20\") " pod="opendatahub/test-curl-7564-139639731676768" Jul 27 17:08:22.338448 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:22.338411 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731676768" Jul 27 17:08:22.489369 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:22.489330 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-139639731676768"] Jul 27 17:08:22.491157 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:08:22.491129 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1c8e34c_0d81_4c39_8fd9_886eb313ad20.slice/crio-145eb5a382a7f95549c17f90484e227646f0f552b108bedeff86ab39bba6aa37 WatchSource:0}: Error finding container 145eb5a382a7f95549c17f90484e227646f0f552b108bedeff86ab39bba6aa37: Status 404 returned error can't find the container with id 145eb5a382a7f95549c17f90484e227646f0f552b108bedeff86ab39bba6aa37 Jul 27 17:08:22.702708 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:22.702681 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731676768" Jul 27 17:08:22.702895 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:22.702721 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d1a5f329406683c3554c054c624ec289887fae7ba9fed9cac95813d180ca5ef" Jul 27 17:08:22.704064 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:22.704039 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-139639731676768" event={"ID":"a1c8e34c-0d81-4c39-8fd9-886eb313ad20","Type":"ContainerStarted","Data":"145eb5a382a7f95549c17f90484e227646f0f552b108bedeff86ab39bba6aa37"} Jul 27 17:08:22.715641 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:22.715613 2550 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7564-139639731676768" oldPodUID="e8eb7a92-9a09-4f8b-8a58-1a926676befb" podUID="a1c8e34c-0d81-4c39-8fd9-886eb313ad20" Jul 27 17:08:23.712530 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:23.712497 2550 generic.go:356] "Generic (PLEG): container finished" podID="a1c8e34c-0d81-4c39-8fd9-886eb313ad20" containerID="f95b8de586f2fce2fa72c746fa699fb828afd91f4c0f108a155eaaabc6df8054" exitCode=0 Jul 27 17:08:23.712935 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:23.712552 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-139639731676768" event={"ID":"a1c8e34c-0d81-4c39-8fd9-886eb313ad20","Type":"ContainerDied","Data":"f95b8de586f2fce2fa72c746fa699fb828afd91f4c0f108a155eaaabc6df8054"} Jul 27 17:08:23.851538 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:23.851497 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8eb7a92-9a09-4f8b-8a58-1a926676befb" path="/var/lib/kubelet/pods/e8eb7a92-9a09-4f8b-8a58-1a926676befb/volumes" Jul 27 17:08:24.879658 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:24.879607 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731676768" Jul 27 17:08:24.901862 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:24.901834 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7564-139639731676768_a1c8e34c-0d81-4c39-8fd9-886eb313ad20/test-curl-7564-139639731676768/0.log" Jul 27 17:08:24.927261 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:24.927229 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7564-139639731676768"] Jul 27 17:08:24.933310 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:24.933283 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7564-139639731676768"] Jul 27 17:08:24.980436 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:24.980356 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5f9m\" (UniqueName: \"kubernetes.io/projected/a1c8e34c-0d81-4c39-8fd9-886eb313ad20-kube-api-access-x5f9m\") pod \"a1c8e34c-0d81-4c39-8fd9-886eb313ad20\" (UID: \"a1c8e34c-0d81-4c39-8fd9-886eb313ad20\") " Jul 27 17:08:24.982630 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:24.982594 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c8e34c-0d81-4c39-8fd9-886eb313ad20-kube-api-access-x5f9m" (OuterVolumeSpecName: "kube-api-access-x5f9m") pod "a1c8e34c-0d81-4c39-8fd9-886eb313ad20" (UID: "a1c8e34c-0d81-4c39-8fd9-886eb313ad20"). InnerVolumeSpecName "kube-api-access-x5f9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:08:25.081804 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:25.081766 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5f9m\" (UniqueName: \"kubernetes.io/projected/a1c8e34c-0d81-4c39-8fd9-886eb313ad20-kube-api-access-x5f9m\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:08:25.728454 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:25.728421 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-139639731676768" Jul 27 17:08:25.728888 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:25.728428 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="145eb5a382a7f95549c17f90484e227646f0f552b108bedeff86ab39bba6aa37" Jul 27 17:08:25.740183 ip-10-0-135-70 kubenswrapper[2550]: E0727 17:08:25.740151 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7564-139639731676768\" is forbidden: User \"system:node:ip-10-0-135-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-70.ec2.internal' and this object" podUID="a1c8e34c-0d81-4c39-8fd9-886eb313ad20" pod="opendatahub/test-curl-7564-139639731676768" Jul 27 17:08:25.851636 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:25.851602 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1c8e34c-0d81-4c39-8fd9-886eb313ad20" path="/var/lib/kubelet/pods/a1c8e34c-0d81-4c39-8fd9-886eb313ad20/volumes" Jul 27 17:08:47.917846 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:47.917807 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf"] Jul 27 17:08:47.918611 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:47.918587 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c8e34c-0d81-4c39-8fd9-886eb313ad20" containerName="test-curl-7564-139639731676768" Jul 27 17:08:47.918611 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:47.918610 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c8e34c-0d81-4c39-8fd9-886eb313ad20" containerName="test-curl-7564-139639731676768" Jul 27 17:08:47.918798 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:47.918711 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c8e34c-0d81-4c39-8fd9-886eb313ad20" containerName="test-curl-7564-139639731676768" Jul 27 17:08:47.922552 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:47.922530 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" Jul 27 17:08:47.925867 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:47.925843 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-de4c74-serving-cert\"" Jul 27 17:08:47.937419 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:47.937391 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf"] Jul 27 17:08:48.005338 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.005301 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fe56b0-95ce-4282-92f0-2f2b188b0941-maas-api-tls\") pod \"maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf\" (UID: \"47fe56b0-95ce-4282-92f0-2f2b188b0941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" Jul 27 17:08:48.005545 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.005383 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xpzb\" (UniqueName: \"kubernetes.io/projected/47fe56b0-95ce-4282-92f0-2f2b188b0941-kube-api-access-4xpzb\") pod \"maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf\" (UID: \"47fe56b0-95ce-4282-92f0-2f2b188b0941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" Jul 27 17:08:48.005545 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.005447 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fe56b0-95ce-4282-92f0-2f2b188b0941-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf\" (UID: \"47fe56b0-95ce-4282-92f0-2f2b188b0941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" Jul 27 17:08:48.106535 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.106498 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xpzb\" (UniqueName: \"kubernetes.io/projected/47fe56b0-95ce-4282-92f0-2f2b188b0941-kube-api-access-4xpzb\") pod \"maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf\" (UID: \"47fe56b0-95ce-4282-92f0-2f2b188b0941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" Jul 27 17:08:48.106725 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.106545 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fe56b0-95ce-4282-92f0-2f2b188b0941-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf\" (UID: \"47fe56b0-95ce-4282-92f0-2f2b188b0941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" Jul 27 17:08:48.106725 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.106617 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fe56b0-95ce-4282-92f0-2f2b188b0941-maas-api-tls\") pod \"maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf\" (UID: \"47fe56b0-95ce-4282-92f0-2f2b188b0941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" Jul 27 17:08:48.106800 ip-10-0-135-70 kubenswrapper[2550]: E0727 17:08:48.106725 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-serving-cert: secret "maas-api-e2e-ipp-b-de4c74-serving-cert" not found Jul 27 17:08:48.106800 ip-10-0-135-70 kubenswrapper[2550]: E0727 17:08:48.106784 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47fe56b0-95ce-4282-92f0-2f2b188b0941-maas-api-tls podName:47fe56b0-95ce-4282-92f0-2f2b188b0941 nodeName:}" failed. No retries permitted until 2026-07-27 17:08:48.606766123 +0000 UTC m=+3107.279300600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47fe56b0-95ce-4282-92f0-2f2b188b0941-maas-api-tls") pod "maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" (UID: "47fe56b0-95ce-4282-92f0-2f2b188b0941") : secret "maas-api-e2e-ipp-b-de4c74-serving-cert" not found Jul 27 17:08:48.107249 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.107230 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fe56b0-95ce-4282-92f0-2f2b188b0941-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf\" (UID: \"47fe56b0-95ce-4282-92f0-2f2b188b0941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" Jul 27 17:08:48.134205 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.134173 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xpzb\" (UniqueName: \"kubernetes.io/projected/47fe56b0-95ce-4282-92f0-2f2b188b0941-kube-api-access-4xpzb\") pod \"maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf\" (UID: \"47fe56b0-95ce-4282-92f0-2f2b188b0941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" Jul 27 17:08:48.611547 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.611500 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fe56b0-95ce-4282-92f0-2f2b188b0941-maas-api-tls\") pod \"maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf\" (UID: \"47fe56b0-95ce-4282-92f0-2f2b188b0941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" Jul 27 17:08:48.614275 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.614241 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fe56b0-95ce-4282-92f0-2f2b188b0941-maas-api-tls\") pod \"maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf\" (UID: \"47fe56b0-95ce-4282-92f0-2f2b188b0941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" Jul 27 17:08:48.689070 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.689032 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth"] Jul 27 17:08:48.693167 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.693141 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth" Jul 27 17:08:48.695806 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.695773 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-de4c74-dockercfg-k68fs\"" Jul 27 17:08:48.696006 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.695777 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-de4c74\"" Jul 27 17:08:48.705269 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.705244 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth"] Jul 27 17:08:48.739341 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.739308 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj"] Jul 27 17:08:48.743158 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.743142 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj" Jul 27 17:08:48.755406 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.755382 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj"] Jul 27 17:08:48.813050 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.813017 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a66dcd8-44de-42ed-b47d-25a474259389-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth\" (UID: \"9a66dcd8-44de-42ed-b47d-25a474259389\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth" Jul 27 17:08:48.813219 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.813053 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31fabae9-f4f2-477b-a721-39f4c73114e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj\" (UID: \"31fabae9-f4f2-477b-a721-39f4c73114e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj" Jul 27 17:08:48.813219 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.813080 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27vr2\" (UniqueName: \"kubernetes.io/projected/31fabae9-f4f2-477b-a721-39f4c73114e9-kube-api-access-27vr2\") pod \"payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj\" (UID: \"31fabae9-f4f2-477b-a721-39f4c73114e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj" Jul 27 17:08:48.813219 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.813172 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd6sw\" (UniqueName: \"kubernetes.io/projected/9a66dcd8-44de-42ed-b47d-25a474259389-kube-api-access-hd6sw\") pod \"payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth\" (UID: \"9a66dcd8-44de-42ed-b47d-25a474259389\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth" Jul 27 17:08:48.838946 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.838893 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" Jul 27 17:08:48.914802 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.914760 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31fabae9-f4f2-477b-a721-39f4c73114e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj\" (UID: \"31fabae9-f4f2-477b-a721-39f4c73114e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj" Jul 27 17:08:48.914997 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.914815 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27vr2\" (UniqueName: \"kubernetes.io/projected/31fabae9-f4f2-477b-a721-39f4c73114e9-kube-api-access-27vr2\") pod \"payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj\" (UID: \"31fabae9-f4f2-477b-a721-39f4c73114e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj" Jul 27 17:08:48.914997 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.914869 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd6sw\" (UniqueName: \"kubernetes.io/projected/9a66dcd8-44de-42ed-b47d-25a474259389-kube-api-access-hd6sw\") pod \"payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth\" (UID: \"9a66dcd8-44de-42ed-b47d-25a474259389\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth" Jul 27 17:08:48.915977 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.915823 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a66dcd8-44de-42ed-b47d-25a474259389-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth\" (UID: \"9a66dcd8-44de-42ed-b47d-25a474259389\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth" Jul 27 17:08:48.915977 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.915845 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31fabae9-f4f2-477b-a721-39f4c73114e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj\" (UID: \"31fabae9-f4f2-477b-a721-39f4c73114e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj" Jul 27 17:08:48.916886 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.916710 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a66dcd8-44de-42ed-b47d-25a474259389-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth\" (UID: \"9a66dcd8-44de-42ed-b47d-25a474259389\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth" Jul 27 17:08:48.928705 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.928671 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd6sw\" (UniqueName: \"kubernetes.io/projected/9a66dcd8-44de-42ed-b47d-25a474259389-kube-api-access-hd6sw\") pod \"payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth\" (UID: \"9a66dcd8-44de-42ed-b47d-25a474259389\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth" Jul 27 17:08:48.930315 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.930294 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27vr2\" (UniqueName: \"kubernetes.io/projected/31fabae9-f4f2-477b-a721-39f4c73114e9-kube-api-access-27vr2\") pod \"payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj\" (UID: \"31fabae9-f4f2-477b-a721-39f4c73114e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj" Jul 27 17:08:48.994027 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:48.993997 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf"] Jul 27 17:08:48.995728 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:08:48.995691 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47fe56b0_95ce_4282_92f0_2f2b188b0941.slice/crio-07f5f507312b75e3fdc1719d4b1de0a2fea03f8fc9f59c1542517a6ee040cecc WatchSource:0}: Error finding container 07f5f507312b75e3fdc1719d4b1de0a2fea03f8fc9f59c1542517a6ee040cecc: Status 404 returned error can't find the container with id 07f5f507312b75e3fdc1719d4b1de0a2fea03f8fc9f59c1542517a6ee040cecc Jul 27 17:08:49.007208 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:49.007181 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth" Jul 27 17:08:49.054741 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:49.054708 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj" Jul 27 17:08:49.169206 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:49.168358 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth"] Jul 27 17:08:49.234643 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:49.234617 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj"] Jul 27 17:08:49.236227 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:08:49.236199 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31fabae9_f4f2_477b_a721_39f4c73114e9.slice/crio-a20abf472989a261cfc9ccd5ba8a5f7133dc364fae2b991a440c488b5ff1c25e WatchSource:0}: Error finding container a20abf472989a261cfc9ccd5ba8a5f7133dc364fae2b991a440c488b5ff1c25e: Status 404 returned error can't find the container with id a20abf472989a261cfc9ccd5ba8a5f7133dc364fae2b991a440c488b5ff1c25e Jul 27 17:08:49.889144 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:49.889107 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj" event={"ID":"31fabae9-f4f2-477b-a721-39f4c73114e9","Type":"ContainerStarted","Data":"13ccc82ab3414c6b6cb4ed8aa61b07d792e29d0e76bae322e1b78a6f2ac76a3c"} Jul 27 17:08:49.889380 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:49.889171 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj" event={"ID":"31fabae9-f4f2-477b-a721-39f4c73114e9","Type":"ContainerStarted","Data":"a20abf472989a261cfc9ccd5ba8a5f7133dc364fae2b991a440c488b5ff1c25e"} Jul 27 17:08:49.889380 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:49.889283 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj" Jul 27 17:08:49.890866 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:49.890834 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" event={"ID":"47fe56b0-95ce-4282-92f0-2f2b188b0941","Type":"ContainerStarted","Data":"7b6575e72d44fbc3ddf210ab8fceb8165c1f115b58b8ed2c8d37c95969dc79ad"} Jul 27 17:08:49.891043 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:49.890891 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" event={"ID":"47fe56b0-95ce-4282-92f0-2f2b188b0941","Type":"ContainerStarted","Data":"07f5f507312b75e3fdc1719d4b1de0a2fea03f8fc9f59c1542517a6ee040cecc"} Jul 27 17:08:49.891043 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:49.890968 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" Jul 27 17:08:49.892488 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:49.892467 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth" event={"ID":"9a66dcd8-44de-42ed-b47d-25a474259389","Type":"ContainerStarted","Data":"d5e420bb123ddae288a7f17c7bb0dfb7f024cce3a0dee17e64d5f8fa2e74d98b"} Jul 27 17:08:49.892562 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:49.892503 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth" event={"ID":"9a66dcd8-44de-42ed-b47d-25a474259389","Type":"ContainerStarted","Data":"a079cf53b54ce4ba73f7bba90de2d687ae5e0c8afa998595bf1cf7e2c369b9d7"} Jul 27 17:08:49.892562 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:49.892545 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth" Jul 27 17:08:49.910981 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:49.910934 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj" podStartSLOduration=1.910920103 podStartE2EDuration="1.910920103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:08:49.909560577 +0000 UTC m=+3108.582095089" watchObservedRunningTime="2026-07-27 17:08:49.910920103 +0000 UTC m=+3108.583454593" Jul 27 17:08:49.929439 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:49.929397 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth" podStartSLOduration=1.9293836770000001 podStartE2EDuration="1.929383677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:08:49.928281185 +0000 UTC m=+3108.600815684" watchObservedRunningTime="2026-07-27 17:08:49.929383677 +0000 UTC m=+3108.601918176" Jul 27 17:08:49.951151 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:49.951101 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" podStartSLOduration=2.6259183 podStartE2EDuration="2.951086586s" totalImagesPullingTime="325.168286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:08:49.946826708 +0000 UTC m=+3108.619361208" watchObservedRunningTime="2026-07-27 17:08:49.951086586 +0000 UTC m=+3108.623621138" Jul 27 17:08:55.907434 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:08:55.907406 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" Jul 27 17:09:00.903821 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:00.903792 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj" Jul 27 17:09:00.904251 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:00.903873 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth" Jul 27 17:09:09.473330 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:09.473303 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj_31fabae9-f4f2-477b-a721-39f4c73114e9/payload-processing/0.log" Jul 27 17:09:19.113285 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:19.113215 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-62ctj"] Jul 27 17:09:19.124083 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:19.124054 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-62ctj"] Jul 27 17:09:19.124257 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:19.124055 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-62ctj" Jul 27 17:09:19.224495 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:19.224461 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drr4s\" (UniqueName: \"kubernetes.io/projected/f00d90e0-7b5a-480a-b10e-d2d00b346a9c-kube-api-access-drr4s\") pod \"kuadrant-operator-catalog-62ctj\" (UID: \"f00d90e0-7b5a-480a-b10e-d2d00b346a9c\") " pod="kuadrant-system/kuadrant-operator-catalog-62ctj" Jul 27 17:09:19.325538 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:19.325499 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drr4s\" (UniqueName: \"kubernetes.io/projected/f00d90e0-7b5a-480a-b10e-d2d00b346a9c-kube-api-access-drr4s\") pod \"kuadrant-operator-catalog-62ctj\" (UID: \"f00d90e0-7b5a-480a-b10e-d2d00b346a9c\") " pod="kuadrant-system/kuadrant-operator-catalog-62ctj" Jul 27 17:09:19.334984 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:19.334951 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drr4s\" (UniqueName: \"kubernetes.io/projected/f00d90e0-7b5a-480a-b10e-d2d00b346a9c-kube-api-access-drr4s\") pod \"kuadrant-operator-catalog-62ctj\" (UID: \"f00d90e0-7b5a-480a-b10e-d2d00b346a9c\") " pod="kuadrant-system/kuadrant-operator-catalog-62ctj" Jul 27 17:09:19.436220 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:19.436137 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-62ctj" Jul 27 17:09:19.584516 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:19.584483 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-62ctj"] Jul 27 17:09:19.585765 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:09:19.585737 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00d90e0_7b5a_480a_b10e_d2d00b346a9c.slice/crio-5cdf11008f107eb7b67965e06fe1643e297095b5473c2405c6db70c3f42891db WatchSource:0}: Error finding container 5cdf11008f107eb7b67965e06fe1643e297095b5473c2405c6db70c3f42891db: Status 404 returned error can't find the container with id 5cdf11008f107eb7b67965e06fe1643e297095b5473c2405c6db70c3f42891db Jul 27 17:09:20.111231 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:20.111195 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-62ctj" event={"ID":"f00d90e0-7b5a-480a-b10e-d2d00b346a9c","Type":"ContainerStarted","Data":"5cdf11008f107eb7b67965e06fe1643e297095b5473c2405c6db70c3f42891db"} Jul 27 17:09:21.118761 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:21.118719 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-62ctj" event={"ID":"f00d90e0-7b5a-480a-b10e-d2d00b346a9c","Type":"ContainerStarted","Data":"be36487bdcdf2b774cf990739f2527a0b9384f6ec3d1d8c5bd01a6877ad98306"} Jul 27 17:09:21.160958 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:21.160883 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-62ctj" podStartSLOduration=1.249958451 podStartE2EDuration="2.160867266s" totalImagesPullingTime="910.908815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:09:21.158383488 +0000 UTC m=+3139.830917987" watchObservedRunningTime="2026-07-27 17:09:21.160867266 +0000 UTC m=+3139.833401843" Jul 27 17:09:29.436611 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:29.436527 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-62ctj" Jul 27 17:09:29.436611 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:29.436590 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-62ctj" Jul 27 17:09:29.459447 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:29.459412 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-62ctj" Jul 27 17:09:30.199688 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:30.199659 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-62ctj" Jul 27 17:09:30.239086 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:30.239046 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-62ctj"] Jul 27 17:09:32.188846 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:32.188803 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-62ctj" podUID="f00d90e0-7b5a-480a-b10e-d2d00b346a9c" containerName="registry-server" containerID="cri-o://be36487bdcdf2b774cf990739f2527a0b9384f6ec3d1d8c5bd01a6877ad98306" gracePeriod=2 Jul 27 17:09:32.457301 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:32.457277 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-62ctj" Jul 27 17:09:32.581083 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:32.581050 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drr4s\" (UniqueName: \"kubernetes.io/projected/f00d90e0-7b5a-480a-b10e-d2d00b346a9c-kube-api-access-drr4s\") pod \"f00d90e0-7b5a-480a-b10e-d2d00b346a9c\" (UID: \"f00d90e0-7b5a-480a-b10e-d2d00b346a9c\") " Jul 27 17:09:32.583336 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:32.583311 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00d90e0-7b5a-480a-b10e-d2d00b346a9c-kube-api-access-drr4s" (OuterVolumeSpecName: "kube-api-access-drr4s") pod "f00d90e0-7b5a-480a-b10e-d2d00b346a9c" (UID: "f00d90e0-7b5a-480a-b10e-d2d00b346a9c"). InnerVolumeSpecName "kube-api-access-drr4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:09:32.682139 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:32.682103 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drr4s\" (UniqueName: \"kubernetes.io/projected/f00d90e0-7b5a-480a-b10e-d2d00b346a9c-kube-api-access-drr4s\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:09:33.197008 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:33.196973 2550 generic.go:356] "Generic (PLEG): container finished" podID="f00d90e0-7b5a-480a-b10e-d2d00b346a9c" containerID="be36487bdcdf2b774cf990739f2527a0b9384f6ec3d1d8c5bd01a6877ad98306" exitCode=0 Jul 27 17:09:33.197486 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:33.197029 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-62ctj" Jul 27 17:09:33.197595 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:33.197033 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-62ctj" event={"ID":"f00d90e0-7b5a-480a-b10e-d2d00b346a9c","Type":"ContainerDied","Data":"be36487bdcdf2b774cf990739f2527a0b9384f6ec3d1d8c5bd01a6877ad98306"} Jul 27 17:09:33.197661 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:33.197635 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-62ctj" event={"ID":"f00d90e0-7b5a-480a-b10e-d2d00b346a9c","Type":"ContainerDied","Data":"5cdf11008f107eb7b67965e06fe1643e297095b5473c2405c6db70c3f42891db"} Jul 27 17:09:33.197725 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:33.197670 2550 scope.go:117] "RemoveContainer" containerID="be36487bdcdf2b774cf990739f2527a0b9384f6ec3d1d8c5bd01a6877ad98306" Jul 27 17:09:33.211930 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:33.211879 2550 scope.go:117] "RemoveContainer" containerID="be36487bdcdf2b774cf990739f2527a0b9384f6ec3d1d8c5bd01a6877ad98306" Jul 27 17:09:33.212226 ip-10-0-135-70 kubenswrapper[2550]: E0727 17:09:33.212205 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be36487bdcdf2b774cf990739f2527a0b9384f6ec3d1d8c5bd01a6877ad98306\": container with ID starting with be36487bdcdf2b774cf990739f2527a0b9384f6ec3d1d8c5bd01a6877ad98306 not found: ID does not exist" containerID="be36487bdcdf2b774cf990739f2527a0b9384f6ec3d1d8c5bd01a6877ad98306" Jul 27 17:09:33.212321 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:33.212234 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be36487bdcdf2b774cf990739f2527a0b9384f6ec3d1d8c5bd01a6877ad98306"} err="failed to get container status \"be36487bdcdf2b774cf990739f2527a0b9384f6ec3d1d8c5bd01a6877ad98306\": rpc error: code = NotFound desc = could not find container \"be36487bdcdf2b774cf990739f2527a0b9384f6ec3d1d8c5bd01a6877ad98306\": container with ID starting with be36487bdcdf2b774cf990739f2527a0b9384f6ec3d1d8c5bd01a6877ad98306 not found: ID does not exist" Jul 27 17:09:33.238456 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:33.238429 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-62ctj"] Jul 27 17:09:33.241258 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:33.241237 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-62ctj"] Jul 27 17:09:33.851245 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:09:33.851204 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f00d90e0-7b5a-480a-b10e-d2d00b346a9c" path="/var/lib/kubelet/pods/f00d90e0-7b5a-480a-b10e-d2d00b346a9c/volumes" Jul 27 17:10:06.898692 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:06.898658 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj_31fabae9-f4f2-477b-a721-39f4c73114e9/payload-processing/0.log" Jul 27 17:10:12.041076 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.041035 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l"] Jul 27 17:10:12.042469 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.042442 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f00d90e0-7b5a-480a-b10e-d2d00b346a9c" containerName="registry-server" Jul 27 17:10:12.042614 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.042604 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f00d90e0-7b5a-480a-b10e-d2d00b346a9c" containerName="registry-server" Jul 27 17:10:12.042820 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.042809 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f00d90e0-7b5a-480a-b10e-d2d00b346a9c" containerName="registry-server" Jul 27 17:10:12.047607 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.047581 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" Jul 27 17:10:12.059823 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.059799 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-7787cf-serving-cert\"" Jul 27 17:10:12.072005 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.071969 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l"] Jul 27 17:10:12.148606 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.148571 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b503488-6e2a-4d41-b9ce-500d92e2caa2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l\" (UID: \"7b503488-6e2a-4d41-b9ce-500d92e2caa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" Jul 27 17:10:12.148801 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.148636 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csvvq\" (UniqueName: \"kubernetes.io/projected/7b503488-6e2a-4d41-b9ce-500d92e2caa2-kube-api-access-csvvq\") pod \"maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l\" (UID: \"7b503488-6e2a-4d41-b9ce-500d92e2caa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" Jul 27 17:10:12.148801 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.148688 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b503488-6e2a-4d41-b9ce-500d92e2caa2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l\" (UID: \"7b503488-6e2a-4d41-b9ce-500d92e2caa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" Jul 27 17:10:12.250185 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.250140 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b503488-6e2a-4d41-b9ce-500d92e2caa2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l\" (UID: \"7b503488-6e2a-4d41-b9ce-500d92e2caa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" Jul 27 17:10:12.250372 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.250222 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csvvq\" (UniqueName: \"kubernetes.io/projected/7b503488-6e2a-4d41-b9ce-500d92e2caa2-kube-api-access-csvvq\") pod \"maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l\" (UID: \"7b503488-6e2a-4d41-b9ce-500d92e2caa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" Jul 27 17:10:12.250372 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.250251 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b503488-6e2a-4d41-b9ce-500d92e2caa2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l\" (UID: \"7b503488-6e2a-4d41-b9ce-500d92e2caa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" Jul 27 17:10:12.250372 ip-10-0-135-70 kubenswrapper[2550]: E0727 17:10:12.250327 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-serving-cert: secret "maas-api-e2e-ipp-cleanup-7787cf-serving-cert" not found Jul 27 17:10:12.250545 ip-10-0-135-70 kubenswrapper[2550]: E0727 17:10:12.250424 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b503488-6e2a-4d41-b9ce-500d92e2caa2-maas-api-tls podName:7b503488-6e2a-4d41-b9ce-500d92e2caa2 nodeName:}" failed. No retries permitted until 2026-07-27 17:10:12.750398535 +0000 UTC m=+3191.422933019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b503488-6e2a-4d41-b9ce-500d92e2caa2-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" (UID: "7b503488-6e2a-4d41-b9ce-500d92e2caa2") : secret "maas-api-e2e-ipp-cleanup-7787cf-serving-cert" not found Jul 27 17:10:12.251013 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.250994 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b503488-6e2a-4d41-b9ce-500d92e2caa2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l\" (UID: \"7b503488-6e2a-4d41-b9ce-500d92e2caa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" Jul 27 17:10:12.266291 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.266253 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csvvq\" (UniqueName: \"kubernetes.io/projected/7b503488-6e2a-4d41-b9ce-500d92e2caa2-kube-api-access-csvvq\") pod \"maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l\" (UID: \"7b503488-6e2a-4d41-b9ce-500d92e2caa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" Jul 27 17:10:12.755010 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.754974 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b503488-6e2a-4d41-b9ce-500d92e2caa2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l\" (UID: \"7b503488-6e2a-4d41-b9ce-500d92e2caa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" Jul 27 17:10:12.757676 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.757648 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b503488-6e2a-4d41-b9ce-500d92e2caa2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l\" (UID: \"7b503488-6e2a-4d41-b9ce-500d92e2caa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" Jul 27 17:10:12.758179 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.758153 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx"] Jul 27 17:10:12.762097 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.762082 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx" Jul 27 17:10:12.766405 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.766378 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-7787cf\"" Jul 27 17:10:12.766544 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.766520 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-7787cf-dockercfg-bd8kd\"" Jul 27 17:10:12.776362 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.776334 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx"] Jul 27 17:10:12.855970 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.855924 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tfs8\" (UniqueName: \"kubernetes.io/projected/fb439378-ec99-4833-8584-67327f4603c9-kube-api-access-5tfs8\") pod \"payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx\" (UID: \"fb439378-ec99-4833-8584-67327f4603c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx" Jul 27 17:10:12.856167 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.856148 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb439378-ec99-4833-8584-67327f4603c9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx\" (UID: \"fb439378-ec99-4833-8584-67327f4603c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx" Jul 27 17:10:12.957076 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.957038 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb439378-ec99-4833-8584-67327f4603c9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx\" (UID: \"fb439378-ec99-4833-8584-67327f4603c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx" Jul 27 17:10:12.957076 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.957077 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tfs8\" (UniqueName: \"kubernetes.io/projected/fb439378-ec99-4833-8584-67327f4603c9-kube-api-access-5tfs8\") pod \"payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx\" (UID: \"fb439378-ec99-4833-8584-67327f4603c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx" Jul 27 17:10:12.957868 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.957834 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb439378-ec99-4833-8584-67327f4603c9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx\" (UID: \"fb439378-ec99-4833-8584-67327f4603c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx" Jul 27 17:10:12.966243 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.966218 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tfs8\" (UniqueName: \"kubernetes.io/projected/fb439378-ec99-4833-8584-67327f4603c9-kube-api-access-5tfs8\") pod \"payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx\" (UID: \"fb439378-ec99-4833-8584-67327f4603c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx" Jul 27 17:10:12.966965 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:12.966951 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" Jul 27 17:10:13.079267 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:13.079232 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx" Jul 27 17:10:13.128156 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:13.128118 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l"] Jul 27 17:10:13.129379 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:10:13.129350 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b503488_6e2a_4d41_b9ce_500d92e2caa2.slice/crio-7c44c4e37954b5726c407a60213adefe990ccde69a1aad74cebfcaba74e26e85 WatchSource:0}: Error finding container 7c44c4e37954b5726c407a60213adefe990ccde69a1aad74cebfcaba74e26e85: Status 404 returned error can't find the container with id 7c44c4e37954b5726c407a60213adefe990ccde69a1aad74cebfcaba74e26e85 Jul 27 17:10:13.237344 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:13.237316 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx"] Jul 27 17:10:13.239402 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:10:13.239375 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb439378_ec99_4833_8584_67327f4603c9.slice/crio-2bd5af99d68cdd23a1644ff9caad3dfbf4b25636e610fa8d979827fd9d49f6bd WatchSource:0}: Error finding container 2bd5af99d68cdd23a1644ff9caad3dfbf4b25636e610fa8d979827fd9d49f6bd: Status 404 returned error can't find the container with id 2bd5af99d68cdd23a1644ff9caad3dfbf4b25636e610fa8d979827fd9d49f6bd Jul 27 17:10:13.469333 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:13.469303 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" event={"ID":"7b503488-6e2a-4d41-b9ce-500d92e2caa2","Type":"ContainerStarted","Data":"7c44c4e37954b5726c407a60213adefe990ccde69a1aad74cebfcaba74e26e85"} Jul 27 17:10:13.470970 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:13.470931 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx" event={"ID":"fb439378-ec99-4833-8584-67327f4603c9","Type":"ContainerStarted","Data":"71c16d8f9b845f9ae14593d47f2ca0b68c8396e59e6513833fcaea3d043fcc2c"} Jul 27 17:10:13.471099 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:13.470984 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx" event={"ID":"fb439378-ec99-4833-8584-67327f4603c9","Type":"ContainerStarted","Data":"2bd5af99d68cdd23a1644ff9caad3dfbf4b25636e610fa8d979827fd9d49f6bd"} Jul 27 17:10:13.471099 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:13.471016 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx" Jul 27 17:10:13.498663 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:13.498599 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx" podStartSLOduration=1.498576301 podStartE2EDuration="1.498576301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:10:13.493946298 +0000 UTC m=+3192.166480790" watchObservedRunningTime="2026-07-27 17:10:13.498576301 +0000 UTC m=+3192.171110801" Jul 27 17:10:14.478637 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:14.478596 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" event={"ID":"7b503488-6e2a-4d41-b9ce-500d92e2caa2","Type":"ContainerStarted","Data":"163808ed9bf5656f4d0503b418f3b4c2e4f59732606355fe626b6189842b7f62"} Jul 27 17:10:14.479041 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:14.478671 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" Jul 27 17:10:14.500506 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:14.500440 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" podStartSLOduration=3.154270613 podStartE2EDuration="3.500417368s" totalImagesPullingTime="346.146755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:10:14.497749861 +0000 UTC m=+3193.170284360" watchObservedRunningTime="2026-07-27 17:10:14.500417368 +0000 UTC m=+3193.172951879" Jul 27 17:10:20.491629 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:20.491600 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" Jul 27 17:10:24.480994 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:24.480963 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx" Jul 27 17:10:30.599244 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:30.599204 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5"] Jul 27 17:10:30.603546 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:30.603522 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5" Jul 27 17:10:30.606730 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:30.606706 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v2ffk\"" Jul 27 17:10:30.612021 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:30.611989 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5"] Jul 27 17:10:30.739370 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:30.739324 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52036edc-9c8b-4554-98bb-8ecdad7cc4cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5\" (UID: \"52036edc-9c8b-4554-98bb-8ecdad7cc4cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5" Jul 27 17:10:30.840467 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:30.840429 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52036edc-9c8b-4554-98bb-8ecdad7cc4cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5\" (UID: \"52036edc-9c8b-4554-98bb-8ecdad7cc4cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5" Jul 27 17:10:30.841118 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:30.841099 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52036edc-9c8b-4554-98bb-8ecdad7cc4cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5\" (UID: \"52036edc-9c8b-4554-98bb-8ecdad7cc4cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5" Jul 27 17:10:30.921586 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:30.921503 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5" Jul 27 17:10:31.075518 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:31.075491 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5"] Jul 27 17:10:31.076516 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:10:31.076491 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52036edc_9c8b_4554_98bb_8ecdad7cc4cf.slice/crio-766e442dab9aec75cba595787b1e952da8a312d2d73fa9509abbd3a3366cfa73 WatchSource:0}: Error finding container 766e442dab9aec75cba595787b1e952da8a312d2d73fa9509abbd3a3366cfa73: Status 404 returned error can't find the container with id 766e442dab9aec75cba595787b1e952da8a312d2d73fa9509abbd3a3366cfa73 Jul 27 17:10:31.593757 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:31.593650 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5" event={"ID":"52036edc-9c8b-4554-98bb-8ecdad7cc4cf","Type":"ContainerStarted","Data":"151101f111877a997eb5108b4247e1c5310fea68c350c5e607e6b78102ee7c35"} Jul 27 17:10:31.593757 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:31.593741 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5" event={"ID":"52036edc-9c8b-4554-98bb-8ecdad7cc4cf","Type":"ContainerStarted","Data":"766e442dab9aec75cba595787b1e952da8a312d2d73fa9509abbd3a3366cfa73"} Jul 27 17:10:31.613774 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:31.613723 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5" podStartSLOduration=1.613705683 podStartE2EDuration="1.613705683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:10:31.611386305 +0000 UTC m=+3210.283920805" watchObservedRunningTime="2026-07-27 17:10:31.613705683 +0000 UTC m=+3210.286240182" Jul 27 17:10:32.602636 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:32.602586 2550 generic.go:356] "Generic (PLEG): container finished" podID="52036edc-9c8b-4554-98bb-8ecdad7cc4cf" containerID="151101f111877a997eb5108b4247e1c5310fea68c350c5e607e6b78102ee7c35" exitCode=0 Jul 27 17:10:32.602854 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:32.602653 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5" event={"ID":"52036edc-9c8b-4554-98bb-8ecdad7cc4cf","Type":"ContainerDied","Data":"151101f111877a997eb5108b4247e1c5310fea68c350c5e607e6b78102ee7c35"} Jul 27 17:10:33.757409 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:33.757386 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5" Jul 27 17:10:33.870533 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:33.870447 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52036edc-9c8b-4554-98bb-8ecdad7cc4cf-maas-api-service-ca\") pod \"52036edc-9c8b-4554-98bb-8ecdad7cc4cf\" (UID: \"52036edc-9c8b-4554-98bb-8ecdad7cc4cf\") " Jul 27 17:10:33.870783 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:33.870762 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52036edc-9c8b-4554-98bb-8ecdad7cc4cf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "52036edc-9c8b-4554-98bb-8ecdad7cc4cf" (UID: "52036edc-9c8b-4554-98bb-8ecdad7cc4cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:10:33.972185 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:33.972143 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52036edc-9c8b-4554-98bb-8ecdad7cc4cf-maas-api-service-ca\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:10:34.618948 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:34.618922 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5" Jul 27 17:10:34.619155 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:34.618950 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5" event={"ID":"52036edc-9c8b-4554-98bb-8ecdad7cc4cf","Type":"ContainerDied","Data":"766e442dab9aec75cba595787b1e952da8a312d2d73fa9509abbd3a3366cfa73"} Jul 27 17:10:34.619155 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:34.618999 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="766e442dab9aec75cba595787b1e952da8a312d2d73fa9509abbd3a3366cfa73" Jul 27 17:10:43.914930 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:43.914260 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l"] Jul 27 17:10:43.914930 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:43.914512 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" podUID="7b503488-6e2a-4d41-b9ce-500d92e2caa2" containerName="maas-api" containerID="cri-o://163808ed9bf5656f4d0503b418f3b4c2e4f59732606355fe626b6189842b7f62" gracePeriod=30 Jul 27 17:10:43.966206 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:43.966163 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx"] Jul 27 17:10:43.966531 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:43.966444 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx" podUID="fb439378-ec99-4833-8584-67327f4603c9" containerName="payload-processing" containerID="cri-o://71c16d8f9b845f9ae14593d47f2ca0b68c8396e59e6513833fcaea3d043fcc2c" gracePeriod=30 Jul 27 17:10:44.232974 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.232944 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" Jul 27 17:10:44.257002 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.256971 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b503488-6e2a-4d41-b9ce-500d92e2caa2-openshift-service-ca\") pod \"7b503488-6e2a-4d41-b9ce-500d92e2caa2\" (UID: \"7b503488-6e2a-4d41-b9ce-500d92e2caa2\") " Jul 27 17:10:44.257158 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.257029 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csvvq\" (UniqueName: \"kubernetes.io/projected/7b503488-6e2a-4d41-b9ce-500d92e2caa2-kube-api-access-csvvq\") pod \"7b503488-6e2a-4d41-b9ce-500d92e2caa2\" (UID: \"7b503488-6e2a-4d41-b9ce-500d92e2caa2\") " Jul 27 17:10:44.257158 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.257067 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b503488-6e2a-4d41-b9ce-500d92e2caa2-maas-api-tls\") pod \"7b503488-6e2a-4d41-b9ce-500d92e2caa2\" (UID: \"7b503488-6e2a-4d41-b9ce-500d92e2caa2\") " Jul 27 17:10:44.257407 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.257381 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b503488-6e2a-4d41-b9ce-500d92e2caa2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b503488-6e2a-4d41-b9ce-500d92e2caa2" (UID: "7b503488-6e2a-4d41-b9ce-500d92e2caa2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:10:44.257561 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.257543 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b503488-6e2a-4d41-b9ce-500d92e2caa2-openshift-service-ca\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:10:44.259587 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.259566 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b503488-6e2a-4d41-b9ce-500d92e2caa2-kube-api-access-csvvq" (OuterVolumeSpecName: "kube-api-access-csvvq") pod "7b503488-6e2a-4d41-b9ce-500d92e2caa2" (UID: "7b503488-6e2a-4d41-b9ce-500d92e2caa2"). InnerVolumeSpecName "kube-api-access-csvvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:10:44.259829 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.259806 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b503488-6e2a-4d41-b9ce-500d92e2caa2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b503488-6e2a-4d41-b9ce-500d92e2caa2" (UID: "7b503488-6e2a-4d41-b9ce-500d92e2caa2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:10:44.263827 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.263810 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx" Jul 27 17:10:44.358781 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.358750 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb439378-ec99-4833-8584-67327f4603c9-plugins-config-volume\") pod \"fb439378-ec99-4833-8584-67327f4603c9\" (UID: \"fb439378-ec99-4833-8584-67327f4603c9\") " Jul 27 17:10:44.358989 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.358840 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tfs8\" (UniqueName: \"kubernetes.io/projected/fb439378-ec99-4833-8584-67327f4603c9-kube-api-access-5tfs8\") pod \"fb439378-ec99-4833-8584-67327f4603c9\" (UID: \"fb439378-ec99-4833-8584-67327f4603c9\") " Jul 27 17:10:44.359071 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.359054 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csvvq\" (UniqueName: \"kubernetes.io/projected/7b503488-6e2a-4d41-b9ce-500d92e2caa2-kube-api-access-csvvq\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:10:44.359143 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.359070 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b503488-6e2a-4d41-b9ce-500d92e2caa2-maas-api-tls\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:10:44.359279 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.359254 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb439378-ec99-4833-8584-67327f4603c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb439378-ec99-4833-8584-67327f4603c9" (UID: "fb439378-ec99-4833-8584-67327f4603c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:10:44.361202 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.361175 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb439378-ec99-4833-8584-67327f4603c9-kube-api-access-5tfs8" (OuterVolumeSpecName: "kube-api-access-5tfs8") pod "fb439378-ec99-4833-8584-67327f4603c9" (UID: "fb439378-ec99-4833-8584-67327f4603c9"). InnerVolumeSpecName "kube-api-access-5tfs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:10:44.460578 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.460499 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb439378-ec99-4833-8584-67327f4603c9-plugins-config-volume\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:10:44.460578 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.460524 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tfs8\" (UniqueName: \"kubernetes.io/projected/fb439378-ec99-4833-8584-67327f4603c9-kube-api-access-5tfs8\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:10:44.689858 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.689826 2550 generic.go:356] "Generic (PLEG): container finished" podID="7b503488-6e2a-4d41-b9ce-500d92e2caa2" containerID="163808ed9bf5656f4d0503b418f3b4c2e4f59732606355fe626b6189842b7f62" exitCode=0 Jul 27 17:10:44.690082 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.689877 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" Jul 27 17:10:44.690082 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.689925 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" event={"ID":"7b503488-6e2a-4d41-b9ce-500d92e2caa2","Type":"ContainerDied","Data":"163808ed9bf5656f4d0503b418f3b4c2e4f59732606355fe626b6189842b7f62"} Jul 27 17:10:44.690082 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.689981 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l" event={"ID":"7b503488-6e2a-4d41-b9ce-500d92e2caa2","Type":"ContainerDied","Data":"7c44c4e37954b5726c407a60213adefe990ccde69a1aad74cebfcaba74e26e85"} Jul 27 17:10:44.690082 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.690015 2550 scope.go:117] "RemoveContainer" containerID="163808ed9bf5656f4d0503b418f3b4c2e4f59732606355fe626b6189842b7f62" Jul 27 17:10:44.691381 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.691359 2550 generic.go:356] "Generic (PLEG): container finished" podID="fb439378-ec99-4833-8584-67327f4603c9" containerID="71c16d8f9b845f9ae14593d47f2ca0b68c8396e59e6513833fcaea3d043fcc2c" exitCode=0 Jul 27 17:10:44.691464 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.691412 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx" Jul 27 17:10:44.691464 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.691420 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx" event={"ID":"fb439378-ec99-4833-8584-67327f4603c9","Type":"ContainerDied","Data":"71c16d8f9b845f9ae14593d47f2ca0b68c8396e59e6513833fcaea3d043fcc2c"} Jul 27 17:10:44.691559 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.691461 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx" event={"ID":"fb439378-ec99-4833-8584-67327f4603c9","Type":"ContainerDied","Data":"2bd5af99d68cdd23a1644ff9caad3dfbf4b25636e610fa8d979827fd9d49f6bd"} Jul 27 17:10:44.703793 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.703772 2550 scope.go:117] "RemoveContainer" containerID="163808ed9bf5656f4d0503b418f3b4c2e4f59732606355fe626b6189842b7f62" Jul 27 17:10:44.704112 ip-10-0-135-70 kubenswrapper[2550]: E0727 17:10:44.704092 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163808ed9bf5656f4d0503b418f3b4c2e4f59732606355fe626b6189842b7f62\": container with ID starting with 163808ed9bf5656f4d0503b418f3b4c2e4f59732606355fe626b6189842b7f62 not found: ID does not exist" containerID="163808ed9bf5656f4d0503b418f3b4c2e4f59732606355fe626b6189842b7f62" Jul 27 17:10:44.704206 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.704119 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163808ed9bf5656f4d0503b418f3b4c2e4f59732606355fe626b6189842b7f62"} err="failed to get container status \"163808ed9bf5656f4d0503b418f3b4c2e4f59732606355fe626b6189842b7f62\": rpc error: code = NotFound desc = could not find container \"163808ed9bf5656f4d0503b418f3b4c2e4f59732606355fe626b6189842b7f62\": container with ID starting with 163808ed9bf5656f4d0503b418f3b4c2e4f59732606355fe626b6189842b7f62 not found: ID does not exist" Jul 27 17:10:44.704206 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.704136 2550 scope.go:117] "RemoveContainer" containerID="71c16d8f9b845f9ae14593d47f2ca0b68c8396e59e6513833fcaea3d043fcc2c" Jul 27 17:10:44.714004 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.713955 2550 scope.go:117] "RemoveContainer" containerID="71c16d8f9b845f9ae14593d47f2ca0b68c8396e59e6513833fcaea3d043fcc2c" Jul 27 17:10:44.714244 ip-10-0-135-70 kubenswrapper[2550]: E0727 17:10:44.714222 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c16d8f9b845f9ae14593d47f2ca0b68c8396e59e6513833fcaea3d043fcc2c\": container with ID starting with 71c16d8f9b845f9ae14593d47f2ca0b68c8396e59e6513833fcaea3d043fcc2c not found: ID does not exist" containerID="71c16d8f9b845f9ae14593d47f2ca0b68c8396e59e6513833fcaea3d043fcc2c" Jul 27 17:10:44.714303 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.714253 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c16d8f9b845f9ae14593d47f2ca0b68c8396e59e6513833fcaea3d043fcc2c"} err="failed to get container status \"71c16d8f9b845f9ae14593d47f2ca0b68c8396e59e6513833fcaea3d043fcc2c\": rpc error: code = NotFound desc = could not find container \"71c16d8f9b845f9ae14593d47f2ca0b68c8396e59e6513833fcaea3d043fcc2c\": container with ID starting with 71c16d8f9b845f9ae14593d47f2ca0b68c8396e59e6513833fcaea3d043fcc2c not found: ID does not exist" Jul 27 17:10:44.717033 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.717012 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l"] Jul 27 17:10:44.721328 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.721309 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7787cf-546bb49548-xbq7l"] Jul 27 17:10:44.733499 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.733468 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx"] Jul 27 17:10:44.736050 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:44.736034 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7787cf-5b6dbd5cb4-9cmrx"] Jul 27 17:10:45.662855 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:45.662819 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5"] Jul 27 17:10:45.665464 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:45.665443 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7787cf-qlpx5"] Jul 27 17:10:45.852330 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:45.852294 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52036edc-9c8b-4554-98bb-8ecdad7cc4cf" path="/var/lib/kubelet/pods/52036edc-9c8b-4554-98bb-8ecdad7cc4cf/volumes" Jul 27 17:10:45.852815 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:45.852793 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b503488-6e2a-4d41-b9ce-500d92e2caa2" path="/var/lib/kubelet/pods/7b503488-6e2a-4d41-b9ce-500d92e2caa2/volumes" Jul 27 17:10:45.853249 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:45.853230 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb439378-ec99-4833-8584-67327f4603c9" path="/var/lib/kubelet/pods/fb439378-ec99-4833-8584-67327f4603c9/volumes" Jul 27 17:10:56.828085 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:56.827998 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x"] Jul 27 17:10:56.828557 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:56.828541 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb439378-ec99-4833-8584-67327f4603c9" containerName="payload-processing" Jul 27 17:10:56.828604 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:56.828560 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb439378-ec99-4833-8584-67327f4603c9" containerName="payload-processing" Jul 27 17:10:56.828604 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:56.828569 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b503488-6e2a-4d41-b9ce-500d92e2caa2" containerName="maas-api" Jul 27 17:10:56.828604 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:56.828575 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b503488-6e2a-4d41-b9ce-500d92e2caa2" containerName="maas-api" Jul 27 17:10:56.828604 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:56.828602 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52036edc-9c8b-4554-98bb-8ecdad7cc4cf" containerName="revoke-keys" Jul 27 17:10:56.828739 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:56.828608 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="52036edc-9c8b-4554-98bb-8ecdad7cc4cf" containerName="revoke-keys" Jul 27 17:10:56.828739 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:56.828675 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="52036edc-9c8b-4554-98bb-8ecdad7cc4cf" containerName="revoke-keys" Jul 27 17:10:56.828739 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:56.828683 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb439378-ec99-4833-8584-67327f4603c9" containerName="payload-processing" Jul 27 17:10:56.828739 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:56.828694 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b503488-6e2a-4d41-b9ce-500d92e2caa2" containerName="maas-api" Jul 27 17:10:56.833046 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:56.833026 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x" Jul 27 17:10:56.835992 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:56.835971 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v2ffk\"" Jul 27 17:10:56.840984 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:56.840924 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x"] Jul 27 17:10:56.895512 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:56.895466 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/680186f4-f9f1-4083-aca6-d3fe7cdb33ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x\" (UID: \"680186f4-f9f1-4083-aca6-d3fe7cdb33ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x" Jul 27 17:10:56.996074 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:56.996029 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/680186f4-f9f1-4083-aca6-d3fe7cdb33ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x\" (UID: \"680186f4-f9f1-4083-aca6-d3fe7cdb33ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x" Jul 27 17:10:56.996640 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:56.996618 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/680186f4-f9f1-4083-aca6-d3fe7cdb33ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x\" (UID: \"680186f4-f9f1-4083-aca6-d3fe7cdb33ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x" Jul 27 17:10:57.146294 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:57.146142 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x" Jul 27 17:10:57.295363 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:57.295326 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x"] Jul 27 17:10:57.296919 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:10:57.296876 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680186f4_f9f1_4083_aca6_d3fe7cdb33ec.slice/crio-390c32e7c0a1971f09252cdfb75a84027b8a3987c7836ab2b1cc2cdf1f00df44 WatchSource:0}: Error finding container 390c32e7c0a1971f09252cdfb75a84027b8a3987c7836ab2b1cc2cdf1f00df44: Status 404 returned error can't find the container with id 390c32e7c0a1971f09252cdfb75a84027b8a3987c7836ab2b1cc2cdf1f00df44 Jul 27 17:10:57.787397 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:57.787360 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x" event={"ID":"680186f4-f9f1-4083-aca6-d3fe7cdb33ec","Type":"ContainerStarted","Data":"8fb0ff9a77168111000ea02f3fff256e0b11695ffb1ae37d74bee642956b1d2c"} Jul 27 17:10:57.787588 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:57.787419 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x" event={"ID":"680186f4-f9f1-4083-aca6-d3fe7cdb33ec","Type":"ContainerStarted","Data":"390c32e7c0a1971f09252cdfb75a84027b8a3987c7836ab2b1cc2cdf1f00df44"} Jul 27 17:10:57.809665 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:57.809604 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x" podStartSLOduration=1.8095833620000001 podStartE2EDuration="1.809583362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:10:57.807947936 +0000 UTC m=+3236.480482432" watchObservedRunningTime="2026-07-27 17:10:57.809583362 +0000 UTC m=+3236.482117861" Jul 27 17:10:58.795207 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:58.795172 2550 generic.go:356] "Generic (PLEG): container finished" podID="680186f4-f9f1-4083-aca6-d3fe7cdb33ec" containerID="8fb0ff9a77168111000ea02f3fff256e0b11695ffb1ae37d74bee642956b1d2c" exitCode=0 Jul 27 17:10:58.795619 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:58.795265 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x" event={"ID":"680186f4-f9f1-4083-aca6-d3fe7cdb33ec","Type":"ContainerDied","Data":"8fb0ff9a77168111000ea02f3fff256e0b11695ffb1ae37d74bee642956b1d2c"} Jul 27 17:10:59.960671 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:10:59.960644 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x" Jul 27 17:11:00.125897 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:11:00.125799 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/680186f4-f9f1-4083-aca6-d3fe7cdb33ec-maas-api-service-ca\") pod \"680186f4-f9f1-4083-aca6-d3fe7cdb33ec\" (UID: \"680186f4-f9f1-4083-aca6-d3fe7cdb33ec\") " Jul 27 17:11:00.126249 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:11:00.126227 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/680186f4-f9f1-4083-aca6-d3fe7cdb33ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "680186f4-f9f1-4083-aca6-d3fe7cdb33ec" (UID: "680186f4-f9f1-4083-aca6-d3fe7cdb33ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:11:00.227552 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:11:00.227518 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/680186f4-f9f1-4083-aca6-d3fe7cdb33ec-maas-api-service-ca\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:11:00.811358 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:11:00.811327 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x" Jul 27 17:11:00.811533 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:11:00.811366 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x" event={"ID":"680186f4-f9f1-4083-aca6-d3fe7cdb33ec","Type":"ContainerDied","Data":"390c32e7c0a1971f09252cdfb75a84027b8a3987c7836ab2b1cc2cdf1f00df44"} Jul 27 17:11:00.811533 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:11:00.811413 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="390c32e7c0a1971f09252cdfb75a84027b8a3987c7836ab2b1cc2cdf1f00df44" Jul 27 17:11:16.925516 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:11:16.925478 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x"] Jul 27 17:11:16.928197 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:11:16.928175 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2cf641-njb4x"] Jul 27 17:11:17.852625 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:11:17.852592 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="680186f4-f9f1-4083-aca6-d3fe7cdb33ec" path="/var/lib/kubelet/pods/680186f4-f9f1-4083-aca6-d3fe7cdb33ec/volumes" Jul 27 17:12:02.154029 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:02.154003 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 17:12:02.164842 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:02.164818 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f9gll_dbedd2ca-c568-40f9-a9d2-be0dae1b803c/ovn-acl-logging/0.log" Jul 27 17:12:07.214931 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:07.214872 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw"] Jul 27 17:12:07.215404 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:07.215388 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="680186f4-f9f1-4083-aca6-d3fe7cdb33ec" containerName="revoke-keys" Jul 27 17:12:07.215447 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:07.215406 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="680186f4-f9f1-4083-aca6-d3fe7cdb33ec" containerName="revoke-keys" Jul 27 17:12:07.215500 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:07.215491 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="680186f4-f9f1-4083-aca6-d3fe7cdb33ec" containerName="revoke-keys" Jul 27 17:12:07.218843 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:07.218823 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw" Jul 27 17:12:07.222891 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:07.222844 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v2ffk\"" Jul 27 17:12:07.227036 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:07.227009 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw"] Jul 27 17:12:07.288014 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:07.287983 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98570b24-c65d-443a-88b9-8db38f606ece-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw\" (UID: \"98570b24-c65d-443a-88b9-8db38f606ece\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw" Jul 27 17:12:07.389547 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:07.389508 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98570b24-c65d-443a-88b9-8db38f606ece-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw\" (UID: \"98570b24-c65d-443a-88b9-8db38f606ece\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw" Jul 27 17:12:07.390159 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:07.390141 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98570b24-c65d-443a-88b9-8db38f606ece-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw\" (UID: \"98570b24-c65d-443a-88b9-8db38f606ece\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw" Jul 27 17:12:07.532509 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:07.532475 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw" Jul 27 17:12:07.684324 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:07.684296 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw"] Jul 27 17:12:07.684839 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:12:07.684816 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98570b24_c65d_443a_88b9_8db38f606ece.slice/crio-15209f7bee2d4057653324fc581a850590ca3818076643afe09e8a76961813c0 WatchSource:0}: Error finding container 15209f7bee2d4057653324fc581a850590ca3818076643afe09e8a76961813c0: Status 404 returned error can't find the container with id 15209f7bee2d4057653324fc581a850590ca3818076643afe09e8a76961813c0 Jul 27 17:12:08.282219 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:08.282175 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw" event={"ID":"98570b24-c65d-443a-88b9-8db38f606ece","Type":"ContainerStarted","Data":"fe504367a601ac40436cc485015613df14eec8e045e31b855334d61578347929"} Jul 27 17:12:08.282621 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:08.282247 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw" event={"ID":"98570b24-c65d-443a-88b9-8db38f606ece","Type":"ContainerStarted","Data":"15209f7bee2d4057653324fc581a850590ca3818076643afe09e8a76961813c0"} Jul 27 17:12:08.302950 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:08.302877 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw" podStartSLOduration=1.302861165 podStartE2EDuration="1.302861165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:12:08.299488648 +0000 UTC m=+3306.972023150" watchObservedRunningTime="2026-07-27 17:12:08.302861165 +0000 UTC m=+3306.975395710" Jul 27 17:12:09.291114 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:09.291083 2550 generic.go:356] "Generic (PLEG): container finished" podID="98570b24-c65d-443a-88b9-8db38f606ece" containerID="fe504367a601ac40436cc485015613df14eec8e045e31b855334d61578347929" exitCode=0 Jul 27 17:12:09.291498 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:09.291149 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw" event={"ID":"98570b24-c65d-443a-88b9-8db38f606ece","Type":"ContainerDied","Data":"fe504367a601ac40436cc485015613df14eec8e045e31b855334d61578347929"} Jul 27 17:12:10.440998 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:10.440967 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw" Jul 27 17:12:10.522159 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:10.522120 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98570b24-c65d-443a-88b9-8db38f606ece-maas-api-service-ca\") pod \"98570b24-c65d-443a-88b9-8db38f606ece\" (UID: \"98570b24-c65d-443a-88b9-8db38f606ece\") " Jul 27 17:12:10.522532 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:10.522508 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98570b24-c65d-443a-88b9-8db38f606ece-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "98570b24-c65d-443a-88b9-8db38f606ece" (UID: "98570b24-c65d-443a-88b9-8db38f606ece"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:12:10.623441 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:10.623360 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98570b24-c65d-443a-88b9-8db38f606ece-maas-api-service-ca\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:12:11.305862 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:11.305828 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw" Jul 27 17:12:11.306089 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:11.305862 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw" event={"ID":"98570b24-c65d-443a-88b9-8db38f606ece","Type":"ContainerDied","Data":"15209f7bee2d4057653324fc581a850590ca3818076643afe09e8a76961813c0"} Jul 27 17:12:11.306089 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:11.305958 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15209f7bee2d4057653324fc581a850590ca3818076643afe09e8a76961813c0" Jul 27 17:12:17.637320 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:17.637284 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf"] Jul 27 17:12:17.637722 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:17.637576 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" podUID="47fe56b0-95ce-4282-92f0-2f2b188b0941" containerName="maas-api" containerID="cri-o://7b6575e72d44fbc3ddf210ab8fceb8165c1f115b58b8ed2c8d37c95969dc79ad" gracePeriod=30 Jul 27 17:12:17.696015 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:17.695980 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj"] Jul 27 17:12:17.696919 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:17.696848 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj" podUID="31fabae9-f4f2-477b-a721-39f4c73114e9" containerName="payload-processing" containerID="cri-o://13ccc82ab3414c6b6cb4ed8aa61b07d792e29d0e76bae322e1b78a6f2ac76a3c" gracePeriod=30 Jul 27 17:12:17.701442 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:17.701404 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth"] Jul 27 17:12:17.702207 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:17.701683 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth" podUID="9a66dcd8-44de-42ed-b47d-25a474259389" containerName="payload-pre-processing" containerID="cri-o://d5e420bb123ddae288a7f17c7bb0dfb7f024cce3a0dee17e64d5f8fa2e74d98b" gracePeriod=30 Jul 27 17:12:17.989031 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:17.989008 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" Jul 27 17:12:17.992356 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:17.992333 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth" Jul 27 17:12:18.005036 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.005013 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj" Jul 27 17:12:18.097603 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.097571 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fe56b0-95ce-4282-92f0-2f2b188b0941-openshift-service-ca\") pod \"47fe56b0-95ce-4282-92f0-2f2b188b0941\" (UID: \"47fe56b0-95ce-4282-92f0-2f2b188b0941\") " Jul 27 17:12:18.097781 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.097631 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27vr2\" (UniqueName: \"kubernetes.io/projected/31fabae9-f4f2-477b-a721-39f4c73114e9-kube-api-access-27vr2\") pod \"31fabae9-f4f2-477b-a721-39f4c73114e9\" (UID: \"31fabae9-f4f2-477b-a721-39f4c73114e9\") " Jul 27 17:12:18.097781 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.097672 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31fabae9-f4f2-477b-a721-39f4c73114e9-plugins-config-volume\") pod \"31fabae9-f4f2-477b-a721-39f4c73114e9\" (UID: \"31fabae9-f4f2-477b-a721-39f4c73114e9\") " Jul 27 17:12:18.097781 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.097702 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xpzb\" (UniqueName: \"kubernetes.io/projected/47fe56b0-95ce-4282-92f0-2f2b188b0941-kube-api-access-4xpzb\") pod \"47fe56b0-95ce-4282-92f0-2f2b188b0941\" (UID: \"47fe56b0-95ce-4282-92f0-2f2b188b0941\") " Jul 27 17:12:18.098042 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.097858 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hd6sw\" (UniqueName: \"kubernetes.io/projected/9a66dcd8-44de-42ed-b47d-25a474259389-kube-api-access-hd6sw\") pod \"9a66dcd8-44de-42ed-b47d-25a474259389\" (UID: \"9a66dcd8-44de-42ed-b47d-25a474259389\") " Jul 27 17:12:18.098042 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.097923 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a66dcd8-44de-42ed-b47d-25a474259389-plugins-config-volume\") pod \"9a66dcd8-44de-42ed-b47d-25a474259389\" (UID: \"9a66dcd8-44de-42ed-b47d-25a474259389\") " Jul 27 17:12:18.098042 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.097979 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fe56b0-95ce-4282-92f0-2f2b188b0941-maas-api-tls\") pod \"47fe56b0-95ce-4282-92f0-2f2b188b0941\" (UID: \"47fe56b0-95ce-4282-92f0-2f2b188b0941\") " Jul 27 17:12:18.098042 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.097998 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47fe56b0-95ce-4282-92f0-2f2b188b0941-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "47fe56b0-95ce-4282-92f0-2f2b188b0941" (UID: "47fe56b0-95ce-4282-92f0-2f2b188b0941"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:12:18.098239 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.098126 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31fabae9-f4f2-477b-a721-39f4c73114e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "31fabae9-f4f2-477b-a721-39f4c73114e9" (UID: "31fabae9-f4f2-477b-a721-39f4c73114e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:12:18.098405 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.098377 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fe56b0-95ce-4282-92f0-2f2b188b0941-openshift-service-ca\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:12:18.098485 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.098412 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31fabae9-f4f2-477b-a721-39f4c73114e9-plugins-config-volume\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:12:18.098485 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.098404 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a66dcd8-44de-42ed-b47d-25a474259389-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9a66dcd8-44de-42ed-b47d-25a474259389" (UID: "9a66dcd8-44de-42ed-b47d-25a474259389"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:12:18.100334 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.100307 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a66dcd8-44de-42ed-b47d-25a474259389-kube-api-access-hd6sw" (OuterVolumeSpecName: "kube-api-access-hd6sw") pod "9a66dcd8-44de-42ed-b47d-25a474259389" (UID: "9a66dcd8-44de-42ed-b47d-25a474259389"). InnerVolumeSpecName "kube-api-access-hd6sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:12:18.100458 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.100342 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31fabae9-f4f2-477b-a721-39f4c73114e9-kube-api-access-27vr2" (OuterVolumeSpecName: "kube-api-access-27vr2") pod "31fabae9-f4f2-477b-a721-39f4c73114e9" (UID: "31fabae9-f4f2-477b-a721-39f4c73114e9"). InnerVolumeSpecName "kube-api-access-27vr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:12:18.100757 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.100737 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47fe56b0-95ce-4282-92f0-2f2b188b0941-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "47fe56b0-95ce-4282-92f0-2f2b188b0941" (UID: "47fe56b0-95ce-4282-92f0-2f2b188b0941"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:12:18.100826 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.100773 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47fe56b0-95ce-4282-92f0-2f2b188b0941-kube-api-access-4xpzb" (OuterVolumeSpecName: "kube-api-access-4xpzb") pod "47fe56b0-95ce-4282-92f0-2f2b188b0941" (UID: "47fe56b0-95ce-4282-92f0-2f2b188b0941"). InnerVolumeSpecName "kube-api-access-4xpzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:12:18.199242 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.199162 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27vr2\" (UniqueName: \"kubernetes.io/projected/31fabae9-f4f2-477b-a721-39f4c73114e9-kube-api-access-27vr2\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:12:18.199242 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.199188 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xpzb\" (UniqueName: \"kubernetes.io/projected/47fe56b0-95ce-4282-92f0-2f2b188b0941-kube-api-access-4xpzb\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:12:18.199242 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.199200 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd6sw\" (UniqueName: \"kubernetes.io/projected/9a66dcd8-44de-42ed-b47d-25a474259389-kube-api-access-hd6sw\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:12:18.199242 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.199208 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a66dcd8-44de-42ed-b47d-25a474259389-plugins-config-volume\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:12:18.199242 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.199217 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fe56b0-95ce-4282-92f0-2f2b188b0941-maas-api-tls\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:12:18.351826 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.351796 2550 generic.go:356] "Generic (PLEG): container finished" podID="31fabae9-f4f2-477b-a721-39f4c73114e9" containerID="13ccc82ab3414c6b6cb4ed8aa61b07d792e29d0e76bae322e1b78a6f2ac76a3c" exitCode=0 Jul 27 17:12:18.352037 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.351839 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj" Jul 27 17:12:18.352037 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.351867 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj" event={"ID":"31fabae9-f4f2-477b-a721-39f4c73114e9","Type":"ContainerDied","Data":"13ccc82ab3414c6b6cb4ed8aa61b07d792e29d0e76bae322e1b78a6f2ac76a3c"} Jul 27 17:12:18.352037 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.351946 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj" event={"ID":"31fabae9-f4f2-477b-a721-39f4c73114e9","Type":"ContainerDied","Data":"a20abf472989a261cfc9ccd5ba8a5f7133dc364fae2b991a440c488b5ff1c25e"} Jul 27 17:12:18.352037 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.351971 2550 scope.go:117] "RemoveContainer" containerID="13ccc82ab3414c6b6cb4ed8aa61b07d792e29d0e76bae322e1b78a6f2ac76a3c" Jul 27 17:12:18.353309 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.353283 2550 generic.go:356] "Generic (PLEG): container finished" podID="47fe56b0-95ce-4282-92f0-2f2b188b0941" containerID="7b6575e72d44fbc3ddf210ab8fceb8165c1f115b58b8ed2c8d37c95969dc79ad" exitCode=0 Jul 27 17:12:18.353422 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.353341 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" Jul 27 17:12:18.353422 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.353371 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" event={"ID":"47fe56b0-95ce-4282-92f0-2f2b188b0941","Type":"ContainerDied","Data":"7b6575e72d44fbc3ddf210ab8fceb8165c1f115b58b8ed2c8d37c95969dc79ad"} Jul 27 17:12:18.353422 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.353415 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf" event={"ID":"47fe56b0-95ce-4282-92f0-2f2b188b0941","Type":"ContainerDied","Data":"07f5f507312b75e3fdc1719d4b1de0a2fea03f8fc9f59c1542517a6ee040cecc"} Jul 27 17:12:18.355062 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.355040 2550 generic.go:356] "Generic (PLEG): container finished" podID="9a66dcd8-44de-42ed-b47d-25a474259389" containerID="d5e420bb123ddae288a7f17c7bb0dfb7f024cce3a0dee17e64d5f8fa2e74d98b" exitCode=0 Jul 27 17:12:18.355183 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.355079 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth" Jul 27 17:12:18.355183 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.355093 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth" event={"ID":"9a66dcd8-44de-42ed-b47d-25a474259389","Type":"ContainerDied","Data":"d5e420bb123ddae288a7f17c7bb0dfb7f024cce3a0dee17e64d5f8fa2e74d98b"} Jul 27 17:12:18.355183 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.355139 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth" event={"ID":"9a66dcd8-44de-42ed-b47d-25a474259389","Type":"ContainerDied","Data":"a079cf53b54ce4ba73f7bba90de2d687ae5e0c8afa998595bf1cf7e2c369b9d7"} Jul 27 17:12:18.364307 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.364289 2550 scope.go:117] "RemoveContainer" containerID="13ccc82ab3414c6b6cb4ed8aa61b07d792e29d0e76bae322e1b78a6f2ac76a3c" Jul 27 17:12:18.364669 ip-10-0-135-70 kubenswrapper[2550]: E0727 17:12:18.364644 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ccc82ab3414c6b6cb4ed8aa61b07d792e29d0e76bae322e1b78a6f2ac76a3c\": container with ID starting with 13ccc82ab3414c6b6cb4ed8aa61b07d792e29d0e76bae322e1b78a6f2ac76a3c not found: ID does not exist" containerID="13ccc82ab3414c6b6cb4ed8aa61b07d792e29d0e76bae322e1b78a6f2ac76a3c" Jul 27 17:12:18.364758 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.364678 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ccc82ab3414c6b6cb4ed8aa61b07d792e29d0e76bae322e1b78a6f2ac76a3c"} err="failed to get container status \"13ccc82ab3414c6b6cb4ed8aa61b07d792e29d0e76bae322e1b78a6f2ac76a3c\": rpc error: code = NotFound desc = could not find container \"13ccc82ab3414c6b6cb4ed8aa61b07d792e29d0e76bae322e1b78a6f2ac76a3c\": container with ID starting with 13ccc82ab3414c6b6cb4ed8aa61b07d792e29d0e76bae322e1b78a6f2ac76a3c not found: ID does not exist" Jul 27 17:12:18.364758 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.364694 2550 scope.go:117] "RemoveContainer" containerID="7b6575e72d44fbc3ddf210ab8fceb8165c1f115b58b8ed2c8d37c95969dc79ad" Jul 27 17:12:18.374202 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.374174 2550 scope.go:117] "RemoveContainer" containerID="7b6575e72d44fbc3ddf210ab8fceb8165c1f115b58b8ed2c8d37c95969dc79ad" Jul 27 17:12:18.374432 ip-10-0-135-70 kubenswrapper[2550]: E0727 17:12:18.374413 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b6575e72d44fbc3ddf210ab8fceb8165c1f115b58b8ed2c8d37c95969dc79ad\": container with ID starting with 7b6575e72d44fbc3ddf210ab8fceb8165c1f115b58b8ed2c8d37c95969dc79ad not found: ID does not exist" containerID="7b6575e72d44fbc3ddf210ab8fceb8165c1f115b58b8ed2c8d37c95969dc79ad" Jul 27 17:12:18.374484 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.374441 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b6575e72d44fbc3ddf210ab8fceb8165c1f115b58b8ed2c8d37c95969dc79ad"} err="failed to get container status \"7b6575e72d44fbc3ddf210ab8fceb8165c1f115b58b8ed2c8d37c95969dc79ad\": rpc error: code = NotFound desc = could not find container \"7b6575e72d44fbc3ddf210ab8fceb8165c1f115b58b8ed2c8d37c95969dc79ad\": container with ID starting with 7b6575e72d44fbc3ddf210ab8fceb8165c1f115b58b8ed2c8d37c95969dc79ad not found: ID does not exist" Jul 27 17:12:18.374484 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.374457 2550 scope.go:117] "RemoveContainer" containerID="d5e420bb123ddae288a7f17c7bb0dfb7f024cce3a0dee17e64d5f8fa2e74d98b" Jul 27 17:12:18.383932 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.383895 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj"] Jul 27 17:12:18.386047 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.386029 2550 scope.go:117] "RemoveContainer" containerID="d5e420bb123ddae288a7f17c7bb0dfb7f024cce3a0dee17e64d5f8fa2e74d98b" Jul 27 17:12:18.386350 ip-10-0-135-70 kubenswrapper[2550]: E0727 17:12:18.386331 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5e420bb123ddae288a7f17c7bb0dfb7f024cce3a0dee17e64d5f8fa2e74d98b\": container with ID starting with d5e420bb123ddae288a7f17c7bb0dfb7f024cce3a0dee17e64d5f8fa2e74d98b not found: ID does not exist" containerID="d5e420bb123ddae288a7f17c7bb0dfb7f024cce3a0dee17e64d5f8fa2e74d98b" Jul 27 17:12:18.386426 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.386355 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5e420bb123ddae288a7f17c7bb0dfb7f024cce3a0dee17e64d5f8fa2e74d98b"} err="failed to get container status \"d5e420bb123ddae288a7f17c7bb0dfb7f024cce3a0dee17e64d5f8fa2e74d98b\": rpc error: code = NotFound desc = could not find container \"d5e420bb123ddae288a7f17c7bb0dfb7f024cce3a0dee17e64d5f8fa2e74d98b\": container with ID starting with d5e420bb123ddae288a7f17c7bb0dfb7f024cce3a0dee17e64d5f8fa2e74d98b not found: ID does not exist" Jul 27 17:12:18.387684 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.387665 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de4c74-5f8d447646-j5nfj"] Jul 27 17:12:18.396578 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.396289 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf"] Jul 27 17:12:18.400927 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.400886 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-de4c74-f6d4b485c-pghxf"] Jul 27 17:12:18.410441 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.410412 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth"] Jul 27 17:12:18.414503 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:18.414474 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de4c74-56d799df96-hklth"] Jul 27 17:12:19.858370 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:19.858335 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31fabae9-f4f2-477b-a721-39f4c73114e9" path="/var/lib/kubelet/pods/31fabae9-f4f2-477b-a721-39f4c73114e9/volumes" Jul 27 17:12:19.858989 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:19.858846 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47fe56b0-95ce-4282-92f0-2f2b188b0941" path="/var/lib/kubelet/pods/47fe56b0-95ce-4282-92f0-2f2b188b0941/volumes" Jul 27 17:12:19.859194 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:19.859180 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a66dcd8-44de-42ed-b47d-25a474259389" path="/var/lib/kubelet/pods/9a66dcd8-44de-42ed-b47d-25a474259389/volumes" Jul 27 17:12:22.290263 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:22.290162 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw"] Jul 27 17:12:22.293186 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:22.293159 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-de4c74-xzpqw"] Jul 27 17:12:23.850990 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:23.850952 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98570b24-c65d-443a-88b9-8db38f606ece" path="/var/lib/kubelet/pods/98570b24-c65d-443a-88b9-8db38f606ece/volumes" Jul 27 17:12:30.003623 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.003581 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f1b699-ltxvt"] Jul 27 17:12:30.004114 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.004100 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47fe56b0-95ce-4282-92f0-2f2b188b0941" containerName="maas-api" Jul 27 17:12:30.004159 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.004116 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="47fe56b0-95ce-4282-92f0-2f2b188b0941" containerName="maas-api" Jul 27 17:12:30.004159 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.004125 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98570b24-c65d-443a-88b9-8db38f606ece" containerName="revoke-keys" Jul 27 17:12:30.004159 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.004130 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="98570b24-c65d-443a-88b9-8db38f606ece" containerName="revoke-keys" Jul 27 17:12:30.004159 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.004157 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a66dcd8-44de-42ed-b47d-25a474259389" containerName="payload-pre-processing" Jul 27 17:12:30.004330 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.004162 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a66dcd8-44de-42ed-b47d-25a474259389" containerName="payload-pre-processing" Jul 27 17:12:30.004330 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.004172 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31fabae9-f4f2-477b-a721-39f4c73114e9" containerName="payload-processing" Jul 27 17:12:30.004330 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.004177 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="31fabae9-f4f2-477b-a721-39f4c73114e9" containerName="payload-processing" Jul 27 17:12:30.004330 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.004254 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="31fabae9-f4f2-477b-a721-39f4c73114e9" containerName="payload-processing" Jul 27 17:12:30.004330 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.004264 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="47fe56b0-95ce-4282-92f0-2f2b188b0941" containerName="maas-api" Jul 27 17:12:30.004330 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.004272 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="98570b24-c65d-443a-88b9-8db38f606ece" containerName="revoke-keys" Jul 27 17:12:30.004330 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.004278 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a66dcd8-44de-42ed-b47d-25a474259389" containerName="payload-pre-processing" Jul 27 17:12:30.009239 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.009219 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f1b699-ltxvt" Jul 27 17:12:30.012057 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.012032 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v2ffk\"" Jul 27 17:12:30.016527 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.016503 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f1b699-ltxvt"] Jul 27 17:12:30.117801 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.117759 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-f1b699-ltxvt\" (UID: \"2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f1b699-ltxvt" Jul 27 17:12:30.218811 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.218765 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-f1b699-ltxvt\" (UID: \"2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f1b699-ltxvt" Jul 27 17:12:30.219444 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.219424 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-f1b699-ltxvt\" (UID: \"2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f1b699-ltxvt" Jul 27 17:12:30.321986 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.321949 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f1b699-ltxvt" Jul 27 17:12:30.471487 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:30.471455 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f1b699-ltxvt"] Jul 27 17:12:30.474754 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:12:30.474727 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc9a68a_dd88_49d3_a1b9_0f7b66e332c2.slice/crio-27f5f289fe3be0cceab4998d2c96b2df0eff5b33ce0c5b303cb9b3e55e6c2ff4 WatchSource:0}: Error finding container 27f5f289fe3be0cceab4998d2c96b2df0eff5b33ce0c5b303cb9b3e55e6c2ff4: Status 404 returned error can't find the container with id 27f5f289fe3be0cceab4998d2c96b2df0eff5b33ce0c5b303cb9b3e55e6c2ff4 Jul 27 17:12:31.450127 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:31.450088 2550 generic.go:356] "Generic (PLEG): container finished" podID="2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2" containerID="3de500cf0ea999dfe483a308db32bd28d0657daadd6d935a2795d48b64cee71c" exitCode=0 Jul 27 17:12:31.450525 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:31.450170 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f1b699-ltxvt" event={"ID":"2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2","Type":"ContainerDied","Data":"3de500cf0ea999dfe483a308db32bd28d0657daadd6d935a2795d48b64cee71c"} Jul 27 17:12:31.450525 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:31.450231 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f1b699-ltxvt" event={"ID":"2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2","Type":"ContainerStarted","Data":"27f5f289fe3be0cceab4998d2c96b2df0eff5b33ce0c5b303cb9b3e55e6c2ff4"} Jul 27 17:12:32.600792 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:32.600759 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f1b699-ltxvt" Jul 27 17:12:32.744991 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:32.744850 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2-maas-api-service-ca\") pod \"2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2\" (UID: \"2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2\") " Jul 27 17:12:32.745259 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:32.745236 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2" (UID: "2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:12:32.846068 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:32.846032 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2-maas-api-service-ca\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:12:33.465352 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:33.465317 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f1b699-ltxvt" Jul 27 17:12:33.465352 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:33.465348 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f1b699-ltxvt" event={"ID":"2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2","Type":"ContainerDied","Data":"27f5f289fe3be0cceab4998d2c96b2df0eff5b33ce0c5b303cb9b3e55e6c2ff4"} Jul 27 17:12:33.465566 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:33.465394 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27f5f289fe3be0cceab4998d2c96b2df0eff5b33ce0c5b303cb9b3e55e6c2ff4" Jul 27 17:12:41.482762 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:41.482727 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f1b699-29752860-tvvkx"] Jul 27 17:12:41.485728 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:41.485701 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f1b699-29752860-tvvkx"] Jul 27 17:12:41.508246 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:41.508214 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7"] Jul 27 17:12:41.509105 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:41.508647 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7" podUID="9549a682-fb22-422c-bae3-1dff76312415" containerName="payload-processing" containerID="cri-o://bdce0e07eda7261ecb3f118f581732dcd4bc3959104772872951819f09e31153" gracePeriod=30 Jul 27 17:12:41.790229 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:41.790204 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7" Jul 27 17:12:41.835718 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:41.835688 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9549a682-fb22-422c-bae3-1dff76312415-plugins-config-volume\") pod \"9549a682-fb22-422c-bae3-1dff76312415\" (UID: \"9549a682-fb22-422c-bae3-1dff76312415\") " Jul 27 17:12:41.835917 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:41.835793 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcfxl\" (UniqueName: \"kubernetes.io/projected/9549a682-fb22-422c-bae3-1dff76312415-kube-api-access-rcfxl\") pod \"9549a682-fb22-422c-bae3-1dff76312415\" (UID: \"9549a682-fb22-422c-bae3-1dff76312415\") " Jul 27 17:12:41.836132 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:41.836107 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9549a682-fb22-422c-bae3-1dff76312415-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9549a682-fb22-422c-bae3-1dff76312415" (UID: "9549a682-fb22-422c-bae3-1dff76312415"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:12:41.836228 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:41.836209 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9549a682-fb22-422c-bae3-1dff76312415-plugins-config-volume\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:12:41.838059 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:41.838035 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9549a682-fb22-422c-bae3-1dff76312415-kube-api-access-rcfxl" (OuterVolumeSpecName: "kube-api-access-rcfxl") pod "9549a682-fb22-422c-bae3-1dff76312415" (UID: "9549a682-fb22-422c-bae3-1dff76312415"). InnerVolumeSpecName "kube-api-access-rcfxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:12:41.851133 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:41.851106 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce42b508-a489-4ca7-beb8-c67c3a4e18fb" path="/var/lib/kubelet/pods/ce42b508-a489-4ca7-beb8-c67c3a4e18fb/volumes" Jul 27 17:12:41.936721 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:41.936684 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcfxl\" (UniqueName: \"kubernetes.io/projected/9549a682-fb22-422c-bae3-1dff76312415-kube-api-access-rcfxl\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:12:42.541376 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:42.541345 2550 generic.go:356] "Generic (PLEG): container finished" podID="9549a682-fb22-422c-bae3-1dff76312415" containerID="bdce0e07eda7261ecb3f118f581732dcd4bc3959104772872951819f09e31153" exitCode=0 Jul 27 17:12:42.541803 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:42.541394 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7" Jul 27 17:12:42.541803 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:42.541416 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7" event={"ID":"9549a682-fb22-422c-bae3-1dff76312415","Type":"ContainerDied","Data":"bdce0e07eda7261ecb3f118f581732dcd4bc3959104772872951819f09e31153"} Jul 27 17:12:42.541803 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:42.541455 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7" event={"ID":"9549a682-fb22-422c-bae3-1dff76312415","Type":"ContainerDied","Data":"088b9ca91330671902a0eab546e7faecc138821349de216954d28a801cf72a0f"} Jul 27 17:12:42.541803 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:42.541475 2550 scope.go:117] "RemoveContainer" containerID="bdce0e07eda7261ecb3f118f581732dcd4bc3959104772872951819f09e31153" Jul 27 17:12:42.552033 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:42.552013 2550 scope.go:117] "RemoveContainer" containerID="bdce0e07eda7261ecb3f118f581732dcd4bc3959104772872951819f09e31153" Jul 27 17:12:42.552304 ip-10-0-135-70 kubenswrapper[2550]: E0727 17:12:42.552284 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdce0e07eda7261ecb3f118f581732dcd4bc3959104772872951819f09e31153\": container with ID starting with bdce0e07eda7261ecb3f118f581732dcd4bc3959104772872951819f09e31153 not found: ID does not exist" containerID="bdce0e07eda7261ecb3f118f581732dcd4bc3959104772872951819f09e31153" Jul 27 17:12:42.552356 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:42.552314 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdce0e07eda7261ecb3f118f581732dcd4bc3959104772872951819f09e31153"} err="failed to get container status \"bdce0e07eda7261ecb3f118f581732dcd4bc3959104772872951819f09e31153\": rpc error: code = NotFound desc = could not find container \"bdce0e07eda7261ecb3f118f581732dcd4bc3959104772872951819f09e31153\": container with ID starting with bdce0e07eda7261ecb3f118f581732dcd4bc3959104772872951819f09e31153 not found: ID does not exist" Jul 27 17:12:42.560169 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:42.560142 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7"] Jul 27 17:12:42.566051 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:42.566015 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f1b699-79f5c465c8-9jfl7"] Jul 27 17:12:43.850858 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:43.850825 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9549a682-fb22-422c-bae3-1dff76312415" path="/var/lib/kubelet/pods/9549a682-fb22-422c-bae3-1dff76312415/volumes" Jul 27 17:12:45.089145 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:45.089109 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f1b699-ltxvt"] Jul 27 17:12:45.094117 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:45.094082 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f1b699-ltxvt"] Jul 27 17:12:45.853278 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:45.853240 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2" path="/var/lib/kubelet/pods/2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2/volumes" Jul 27 17:12:52.341825 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:52.341780 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7"] Jul 27 17:12:52.342327 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:52.342311 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2" containerName="revoke-keys" Jul 27 17:12:52.342369 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:52.342328 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2" containerName="revoke-keys" Jul 27 17:12:52.342369 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:52.342346 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9549a682-fb22-422c-bae3-1dff76312415" containerName="payload-processing" Jul 27 17:12:52.342369 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:52.342352 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9549a682-fb22-422c-bae3-1dff76312415" containerName="payload-processing" Jul 27 17:12:52.342470 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:52.342430 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9549a682-fb22-422c-bae3-1dff76312415" containerName="payload-processing" Jul 27 17:12:52.342470 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:52.342439 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bc9a68a-dd88-49d3-a1b9-0f7b66e332c2" containerName="revoke-keys" Jul 27 17:12:52.346775 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:52.346759 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7" Jul 27 17:12:52.349793 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:52.349769 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v2ffk\"" Jul 27 17:12:52.353731 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:52.353705 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7"] Jul 27 17:12:52.436797 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:52.436765 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9130744c-7ec7-42bb-9dc8-2a723263b087-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7\" (UID: \"9130744c-7ec7-42bb-9dc8-2a723263b087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7" Jul 27 17:12:52.537471 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:52.537429 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9130744c-7ec7-42bb-9dc8-2a723263b087-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7\" (UID: \"9130744c-7ec7-42bb-9dc8-2a723263b087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7" Jul 27 17:12:52.538114 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:52.538097 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9130744c-7ec7-42bb-9dc8-2a723263b087-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7\" (UID: \"9130744c-7ec7-42bb-9dc8-2a723263b087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7" Jul 27 17:12:52.660073 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:52.659989 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7" Jul 27 17:12:52.807437 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:52.807404 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7"] Jul 27 17:12:52.808379 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:12:52.808350 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9130744c_7ec7_42bb_9dc8_2a723263b087.slice/crio-c0e8edaa136e77403ed969c7b2b10eec1abe51964911a0d1c19770092f290ff7 WatchSource:0}: Error finding container c0e8edaa136e77403ed969c7b2b10eec1abe51964911a0d1c19770092f290ff7: Status 404 returned error can't find the container with id c0e8edaa136e77403ed969c7b2b10eec1abe51964911a0d1c19770092f290ff7 Jul 27 17:12:53.618735 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:53.618646 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7" event={"ID":"9130744c-7ec7-42bb-9dc8-2a723263b087","Type":"ContainerStarted","Data":"1edbb4bb927e80eb1a148c61dcfb59c3dbc50e3ef27d243964843d2e3cbb4a93"} Jul 27 17:12:53.619195 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:53.618738 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7" event={"ID":"9130744c-7ec7-42bb-9dc8-2a723263b087","Type":"ContainerStarted","Data":"c0e8edaa136e77403ed969c7b2b10eec1abe51964911a0d1c19770092f290ff7"} Jul 27 17:12:53.635286 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:53.635231 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7" podStartSLOduration=1.635212309 podStartE2EDuration="1.635212309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:12:53.635013277 +0000 UTC m=+3352.307547776" watchObservedRunningTime="2026-07-27 17:12:53.635212309 +0000 UTC m=+3352.307746808" Jul 27 17:12:54.627164 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:54.627125 2550 generic.go:356] "Generic (PLEG): container finished" podID="9130744c-7ec7-42bb-9dc8-2a723263b087" containerID="1edbb4bb927e80eb1a148c61dcfb59c3dbc50e3ef27d243964843d2e3cbb4a93" exitCode=0 Jul 27 17:12:54.627573 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:54.627205 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7" event={"ID":"9130744c-7ec7-42bb-9dc8-2a723263b087","Type":"ContainerDied","Data":"1edbb4bb927e80eb1a148c61dcfb59c3dbc50e3ef27d243964843d2e3cbb4a93"} Jul 27 17:12:55.780712 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:55.780687 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7" Jul 27 17:12:55.872639 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:55.872613 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9130744c-7ec7-42bb-9dc8-2a723263b087-maas-api-service-ca\") pod \"9130744c-7ec7-42bb-9dc8-2a723263b087\" (UID: \"9130744c-7ec7-42bb-9dc8-2a723263b087\") " Jul 27 17:12:55.873006 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:55.872986 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9130744c-7ec7-42bb-9dc8-2a723263b087-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9130744c-7ec7-42bb-9dc8-2a723263b087" (UID: "9130744c-7ec7-42bb-9dc8-2a723263b087"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:12:55.974412 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:55.974313 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9130744c-7ec7-42bb-9dc8-2a723263b087-maas-api-service-ca\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:12:56.641782 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:56.641753 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7" Jul 27 17:12:56.641951 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:56.641796 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7" event={"ID":"9130744c-7ec7-42bb-9dc8-2a723263b087","Type":"ContainerDied","Data":"c0e8edaa136e77403ed969c7b2b10eec1abe51964911a0d1c19770092f290ff7"} Jul 27 17:12:56.641951 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:12:56.641842 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0e8edaa136e77403ed969c7b2b10eec1abe51964911a0d1c19770092f290ff7" Jul 27 17:13:02.509833 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:02.509798 2550 scope.go:117] "RemoveContainer" containerID="88c73010b9df3af0e83a2f2e2f1db9e1168ffb1b2e1e9e90bed9f35d1983dd7a" Jul 27 17:13:03.193143 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.193099 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d4529a-29752860-cbjkh"] Jul 27 17:13:03.200597 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.200560 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d4529a-29752860-cbjkh"] Jul 27 17:13:03.238810 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.238774 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn"] Jul 27 17:13:03.239071 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.239043 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn" podUID="cf24f376-193e-4fab-87a4-f89c13bca1a0" containerName="payload-pre-processing" containerID="cri-o://de01ab09bb94ba361ba7bd519deece9bd8b1b1808ea8ba18a34df90458d12c9b" gracePeriod=30 Jul 27 17:13:03.496560 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.496537 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn" Jul 27 17:13:03.656248 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.656212 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8fbb\" (UniqueName: \"kubernetes.io/projected/cf24f376-193e-4fab-87a4-f89c13bca1a0-kube-api-access-v8fbb\") pod \"cf24f376-193e-4fab-87a4-f89c13bca1a0\" (UID: \"cf24f376-193e-4fab-87a4-f89c13bca1a0\") " Jul 27 17:13:03.656647 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.656326 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf24f376-193e-4fab-87a4-f89c13bca1a0-plugins-config-volume\") pod \"cf24f376-193e-4fab-87a4-f89c13bca1a0\" (UID: \"cf24f376-193e-4fab-87a4-f89c13bca1a0\") " Jul 27 17:13:03.656740 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.656720 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf24f376-193e-4fab-87a4-f89c13bca1a0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cf24f376-193e-4fab-87a4-f89c13bca1a0" (UID: "cf24f376-193e-4fab-87a4-f89c13bca1a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:13:03.658488 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.658463 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf24f376-193e-4fab-87a4-f89c13bca1a0-kube-api-access-v8fbb" (OuterVolumeSpecName: "kube-api-access-v8fbb") pod "cf24f376-193e-4fab-87a4-f89c13bca1a0" (UID: "cf24f376-193e-4fab-87a4-f89c13bca1a0"). InnerVolumeSpecName "kube-api-access-v8fbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:13:03.688980 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.688942 2550 generic.go:356] "Generic (PLEG): container finished" podID="cf24f376-193e-4fab-87a4-f89c13bca1a0" containerID="de01ab09bb94ba361ba7bd519deece9bd8b1b1808ea8ba18a34df90458d12c9b" exitCode=0 Jul 27 17:13:03.689145 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.688991 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn" Jul 27 17:13:03.689145 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.689029 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn" event={"ID":"cf24f376-193e-4fab-87a4-f89c13bca1a0","Type":"ContainerDied","Data":"de01ab09bb94ba361ba7bd519deece9bd8b1b1808ea8ba18a34df90458d12c9b"} Jul 27 17:13:03.689145 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.689086 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn" event={"ID":"cf24f376-193e-4fab-87a4-f89c13bca1a0","Type":"ContainerDied","Data":"7fa7ea3b1fe55021b4d26ef844aa0fc5a6d1f7d0ba4b2c8c968da53ecab37ebb"} Jul 27 17:13:03.689145 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.689110 2550 scope.go:117] "RemoveContainer" containerID="de01ab09bb94ba361ba7bd519deece9bd8b1b1808ea8ba18a34df90458d12c9b" Jul 27 17:13:03.709010 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.708988 2550 scope.go:117] "RemoveContainer" containerID="de01ab09bb94ba361ba7bd519deece9bd8b1b1808ea8ba18a34df90458d12c9b" Jul 27 17:13:03.709321 ip-10-0-135-70 kubenswrapper[2550]: E0727 17:13:03.709303 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de01ab09bb94ba361ba7bd519deece9bd8b1b1808ea8ba18a34df90458d12c9b\": container with ID starting with de01ab09bb94ba361ba7bd519deece9bd8b1b1808ea8ba18a34df90458d12c9b not found: ID does not exist" containerID="de01ab09bb94ba361ba7bd519deece9bd8b1b1808ea8ba18a34df90458d12c9b" Jul 27 17:13:03.709425 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.709331 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de01ab09bb94ba361ba7bd519deece9bd8b1b1808ea8ba18a34df90458d12c9b"} err="failed to get container status \"de01ab09bb94ba361ba7bd519deece9bd8b1b1808ea8ba18a34df90458d12c9b\": rpc error: code = NotFound desc = could not find container \"de01ab09bb94ba361ba7bd519deece9bd8b1b1808ea8ba18a34df90458d12c9b\": container with ID starting with de01ab09bb94ba361ba7bd519deece9bd8b1b1808ea8ba18a34df90458d12c9b not found: ID does not exist" Jul 27 17:13:03.722771 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.722704 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn"] Jul 27 17:13:03.726475 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.726454 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d4529a-7c69dc96b4-qk2pn"] Jul 27 17:13:03.757768 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.757738 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8fbb\" (UniqueName: \"kubernetes.io/projected/cf24f376-193e-4fab-87a4-f89c13bca1a0-kube-api-access-v8fbb\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:13:03.757768 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.757767 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf24f376-193e-4fab-87a4-f89c13bca1a0-plugins-config-volume\") on node \"ip-10-0-135-70.ec2.internal\" DevicePath \"\"" Jul 27 17:13:03.851259 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.851225 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a62432d5-2d94-421a-a505-5c319ff0c8f2" path="/var/lib/kubelet/pods/a62432d5-2d94-421a-a505-5c319ff0c8f2/volumes" Jul 27 17:13:03.851569 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:03.851556 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf24f376-193e-4fab-87a4-f89c13bca1a0" path="/var/lib/kubelet/pods/cf24f376-193e-4fab-87a4-f89c13bca1a0/volumes" Jul 27 17:13:07.410856 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:07.410819 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7"] Jul 27 17:13:07.413590 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:07.413566 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d4529a-zfmk7"] Jul 27 17:13:07.850302 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:07.850266 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9130744c-7ec7-42bb-9dc8-2a723263b087" path="/var/lib/kubelet/pods/9130744c-7ec7-42bb-9dc8-2a723263b087/volumes" Jul 27 17:13:14.844524 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:14.844489 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-dc575bd5f-kwdkq_cf79249e-3925-4569-ab1c-7944f8099652/authorino/0.log" Jul 27 17:13:20.256860 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:20.256823 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-74lck_c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88/manager/0.log" Jul 27 17:13:20.377817 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:20.377780 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-cpc94_172833a1-d9d0-41f5-86a8-cf227d9a7479/manager/0.log" Jul 27 17:13:20.861165 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:20.861132 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55548c94b5-9zdpp_2730b6a9-a99f-4a61-b69c-74d48ee48c74/copy-manifests/0.log" Jul 27 17:13:20.871384 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:20.871355 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55548c94b5-9zdpp_2730b6a9-a99f-4a61-b69c-74d48ee48c74/manager/0.log" Jul 27 17:13:21.492253 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:21.492225 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-c7777c89c-g8vbl_b5f3323a-4152-4078-b02f-b8e7ead3e441/maas-api/0.log" Jul 27 17:13:21.829914 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:21.829879 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29752860-l2mzl_2b9fd0a8-4ecc-4fc1-b035-0e0bec728aab/cleanup/0.log" Jul 27 17:13:21.939767 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:21.939740 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-9hkwj_7b0eeb5c-d0f8-4573-b163-c2a54d1fbb54/postgres/0.log" Jul 27 17:13:22.707122 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:22.707095 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p_f55fb6fb-63db-4382-bb60-1ac2a9bc7a14/util/0.log" Jul 27 17:13:22.714117 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:22.714087 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p_f55fb6fb-63db-4382-bb60-1ac2a9bc7a14/pull/0.log" Jul 27 17:13:22.720371 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:22.720344 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p_f55fb6fb-63db-4382-bb60-1ac2a9bc7a14/extract/0.log" Jul 27 17:13:22.831258 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:22.831226 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl_e6fd183d-09c5-43b3-894e-99a5c0c4c6d4/util/0.log" Jul 27 17:13:22.838447 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:22.838421 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl_e6fd183d-09c5-43b3-894e-99a5c0c4c6d4/pull/0.log" Jul 27 17:13:22.845430 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:22.845401 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl_e6fd183d-09c5-43b3-894e-99a5c0c4c6d4/extract/0.log" Jul 27 17:13:22.958810 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:22.958717 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz_fe048780-6d3f-4ba5-b296-ef02c3fee70c/util/0.log" Jul 27 17:13:22.965400 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:22.965370 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz_fe048780-6d3f-4ba5-b296-ef02c3fee70c/pull/0.log" Jul 27 17:13:22.971438 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:22.971412 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz_fe048780-6d3f-4ba5-b296-ef02c3fee70c/extract/0.log" Jul 27 17:13:23.083644 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:23.083618 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm_25a355db-00df-4c8f-82f1-13a486427ea3/util/0.log" Jul 27 17:13:23.090460 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:23.090428 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm_25a355db-00df-4c8f-82f1-13a486427ea3/pull/0.log" Jul 27 17:13:23.097226 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:23.097198 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm_25a355db-00df-4c8f-82f1-13a486427ea3/extract/0.log" Jul 27 17:13:23.225138 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:23.225042 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-dc575bd5f-kwdkq_cf79249e-3925-4569-ab1c-7944f8099652/authorino/0.log" Jul 27 17:13:23.341047 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:23.341015 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-kj6pt_a236c9ad-7456-4935-bc28-9dc31aeaf99a/manager/0.log" Jul 27 17:13:23.457090 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:23.457055 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-njb5q_feaa1fea-b20f-41e4-b290-35f7f9b7bdda/manager/0.log" Jul 27 17:13:23.572966 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:23.572937 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-dhx4s_f67521c8-88be-43e7-9136-c7817310801f/kuadrant-console-plugin/0.log" Jul 27 17:13:23.698443 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:23.698416 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-sckj9_f3122b10-967f-4583-bdc0-9fb1e8ce81e5/registry-server/0.log" Jul 27 17:13:23.841571 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:23.841489 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-9s62w_86e77d7e-9cb7-468e-af10-06145a7a68c7/manager/0.log" Jul 27 17:13:24.078877 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:24.078840 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-mj5zs_83affb8e-8575-4e39-875b-7bd4990a87e3/manager/0.log" Jul 27 17:13:24.431501 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:24.431466 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799czw46d_022c0a96-b89f-484b-a715-6b90cc6cc408/istio-proxy/0.log" Jul 27 17:13:24.773623 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:24.773599 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78dbfdf94f-9trzp_31084f38-f413-40bf-a8ac-e5eaaa47b33a/kube-auth-proxy/0.log" Jul 27 17:13:25.020074 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:25.020041 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-kx6r6_d6e2c3b5-1303-4a7c-b111-2b0b661ff60c/payload-pre-processing/0.log" Jul 27 17:13:25.604599 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:25.604570 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-tqh65_a0e3481c-db1e-443d-af83-e7b0e46a1be0/main/0.log" Jul 27 17:13:25.848517 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:25.848487 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-rmmz4_fb3eaca1-fe1c-47c8-b733-50dbcba620c2/main/0.log" Jul 27 17:13:35.028635 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:35.028599 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lkclq_efd59b32-c96c-4808-9a83-b24cdc2e3dc7/global-pull-secret-syncer/0.log" Jul 27 17:13:35.234498 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:35.234466 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tzvpv_5445bc4e-d01a-48c8-8c71-6541ad9a487e/konnectivity-agent/0.log" Jul 27 17:13:35.303894 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:35.303804 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-70.ec2.internal_efedf659439c0f4b07ff2be7c9f4e1bd/haproxy/0.log" Jul 27 17:13:40.030753 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:40.030724 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p_f55fb6fb-63db-4382-bb60-1ac2a9bc7a14/extract/0.log" Jul 27 17:13:40.057503 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:40.057472 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p_f55fb6fb-63db-4382-bb60-1ac2a9bc7a14/util/0.log" Jul 27 17:13:40.087261 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:40.087230 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wqh9p_f55fb6fb-63db-4382-bb60-1ac2a9bc7a14/pull/0.log" Jul 27 17:13:40.140142 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:40.140112 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl_e6fd183d-09c5-43b3-894e-99a5c0c4c6d4/extract/0.log" Jul 27 17:13:40.164310 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:40.164283 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl_e6fd183d-09c5-43b3-894e-99a5c0c4c6d4/util/0.log" Jul 27 17:13:40.191312 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:40.191287 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxtsl_e6fd183d-09c5-43b3-894e-99a5c0c4c6d4/pull/0.log" Jul 27 17:13:40.242132 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:40.242102 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz_fe048780-6d3f-4ba5-b296-ef02c3fee70c/extract/0.log" Jul 27 17:13:40.277564 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:40.277540 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz_fe048780-6d3f-4ba5-b296-ef02c3fee70c/util/0.log" Jul 27 17:13:40.300069 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:40.299994 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pmvcz_fe048780-6d3f-4ba5-b296-ef02c3fee70c/pull/0.log" Jul 27 17:13:40.326551 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:40.326522 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm_25a355db-00df-4c8f-82f1-13a486427ea3/extract/0.log" Jul 27 17:13:40.349096 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:40.349067 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm_25a355db-00df-4c8f-82f1-13a486427ea3/util/0.log" Jul 27 17:13:40.376263 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:40.376234 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nscsm_25a355db-00df-4c8f-82f1-13a486427ea3/pull/0.log" Jul 27 17:13:40.958546 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:40.958514 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-dc575bd5f-kwdkq_cf79249e-3925-4569-ab1c-7944f8099652/authorino/0.log" Jul 27 17:13:40.988853 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:40.988827 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-kj6pt_a236c9ad-7456-4935-bc28-9dc31aeaf99a/manager/0.log" Jul 27 17:13:41.017213 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:41.017187 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-njb5q_feaa1fea-b20f-41e4-b290-35f7f9b7bdda/manager/0.log" Jul 27 17:13:41.050403 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:41.050378 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-dhx4s_f67521c8-88be-43e7-9136-c7817310801f/kuadrant-console-plugin/0.log" Jul 27 17:13:41.114874 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:41.114844 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-sckj9_f3122b10-967f-4583-bdc0-9fb1e8ce81e5/registry-server/0.log" Jul 27 17:13:41.336212 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:41.336169 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-9s62w_86e77d7e-9cb7-468e-af10-06145a7a68c7/manager/0.log" Jul 27 17:13:41.503685 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:41.503652 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-mj5zs_83affb8e-8575-4e39-875b-7bd4990a87e3/manager/0.log" Jul 27 17:13:43.250358 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:43.250327 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15eefb0f-0903-456a-a77e-5608f909512b/alertmanager/0.log" Jul 27 17:13:43.295738 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:43.295702 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15eefb0f-0903-456a-a77e-5608f909512b/config-reloader/0.log" Jul 27 17:13:43.346404 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:43.346373 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15eefb0f-0903-456a-a77e-5608f909512b/kube-rbac-proxy-web/0.log" Jul 27 17:13:43.385153 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:43.385118 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15eefb0f-0903-456a-a77e-5608f909512b/kube-rbac-proxy/0.log" Jul 27 17:13:43.412216 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:43.412190 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15eefb0f-0903-456a-a77e-5608f909512b/kube-rbac-proxy-metric/0.log" Jul 27 17:13:43.440384 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:43.440359 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15eefb0f-0903-456a-a77e-5608f909512b/prom-label-proxy/0.log" Jul 27 17:13:43.472019 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:43.471993 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15eefb0f-0903-456a-a77e-5608f909512b/init-config-reloader/0.log" Jul 27 17:13:43.530742 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:43.530713 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-4skw2_2528af38-f6e1-4c69-859a-6d00a5301ce4/cluster-monitoring-operator/0.log" Jul 27 17:13:43.672591 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:43.672560 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-76d9d7c7f8-5rn2l_6c20cd6a-6b94-4fe7-8bf7-ee7b00fd60c9/metrics-server/0.log" Jul 27 17:13:43.713078 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:43.713047 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-x9964_6a364884-ca85-47df-9f19-b2789c22c312/monitoring-plugin/0.log" Jul 27 17:13:43.978944 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:43.978828 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vndgf_60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090/node-exporter/0.log" Jul 27 17:13:44.015491 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:44.015462 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vndgf_60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090/kube-rbac-proxy/0.log" Jul 27 17:13:44.041821 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:44.041794 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vndgf_60b9b3b2-5ab3-4bcc-a17a-ff64fcbe4090/init-textfile/0.log" Jul 27 17:13:44.403267 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:44.403236 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-gcx88_5dba214b-8b12-4185-af7b-c5cc09216918/prometheus-operator/0.log" Jul 27 17:13:44.423963 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:44.423927 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-gcx88_5dba214b-8b12-4185-af7b-c5cc09216918/kube-rbac-proxy/0.log" Jul 27 17:13:44.451221 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:44.451195 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-8tbkw_9b5e5814-4c80-42ab-88d8-593e29b22606/prometheus-operator-admission-webhook/0.log" Jul 27 17:13:44.572454 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:44.572426 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6456df978d-5vg66_3e3b341d-cd31-447e-b15c-04289beef65b/thanos-query/0.log" Jul 27 17:13:44.594746 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:44.594718 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6456df978d-5vg66_3e3b341d-cd31-447e-b15c-04289beef65b/kube-rbac-proxy-web/0.log" Jul 27 17:13:44.619797 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:44.619761 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6456df978d-5vg66_3e3b341d-cd31-447e-b15c-04289beef65b/kube-rbac-proxy/0.log" Jul 27 17:13:44.643812 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:44.643780 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6456df978d-5vg66_3e3b341d-cd31-447e-b15c-04289beef65b/prom-label-proxy/0.log" Jul 27 17:13:44.667939 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:44.667863 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6456df978d-5vg66_3e3b341d-cd31-447e-b15c-04289beef65b/kube-rbac-proxy-rules/0.log" Jul 27 17:13:44.700343 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:44.700316 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6456df978d-5vg66_3e3b341d-cd31-447e-b15c-04289beef65b/kube-rbac-proxy-metrics/0.log" Jul 27 17:13:45.269847 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.269813 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw"] Jul 27 17:13:45.270327 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.270311 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9130744c-7ec7-42bb-9dc8-2a723263b087" containerName="revoke-keys" Jul 27 17:13:45.270375 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.270328 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9130744c-7ec7-42bb-9dc8-2a723263b087" containerName="revoke-keys" Jul 27 17:13:45.270375 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.270347 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf24f376-193e-4fab-87a4-f89c13bca1a0" containerName="payload-pre-processing" Jul 27 17:13:45.270375 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.270353 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf24f376-193e-4fab-87a4-f89c13bca1a0" containerName="payload-pre-processing" Jul 27 17:13:45.270466 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.270412 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf24f376-193e-4fab-87a4-f89c13bca1a0" containerName="payload-pre-processing" Jul 27 17:13:45.270466 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.270421 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9130744c-7ec7-42bb-9dc8-2a723263b087" containerName="revoke-keys" Jul 27 17:13:45.273598 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.273583 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" Jul 27 17:13:45.277450 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.277431 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ddw25\"/\"kube-root-ca.crt\"" Jul 27 17:13:45.277742 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.277712 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ddw25\"/\"openshift-service-ca.crt\"" Jul 27 17:13:45.278705 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.278689 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ddw25\"/\"default-dockercfg-nqbtv\"" Jul 27 17:13:45.292699 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.292675 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw"] Jul 27 17:13:45.368551 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.368519 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhqqq\" (UniqueName: \"kubernetes.io/projected/0e2cf884-5d60-4e9b-9410-d8f424c2a1ee-kube-api-access-nhqqq\") pod \"perf-node-gather-daemonset-hqrfw\" (UID: \"0e2cf884-5d60-4e9b-9410-d8f424c2a1ee\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" Jul 27 17:13:45.368693 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.368569 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e2cf884-5d60-4e9b-9410-d8f424c2a1ee-podres\") pod \"perf-node-gather-daemonset-hqrfw\" (UID: \"0e2cf884-5d60-4e9b-9410-d8f424c2a1ee\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" Jul 27 17:13:45.368693 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.368616 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e2cf884-5d60-4e9b-9410-d8f424c2a1ee-sys\") pod \"perf-node-gather-daemonset-hqrfw\" (UID: \"0e2cf884-5d60-4e9b-9410-d8f424c2a1ee\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" Jul 27 17:13:45.368693 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.368689 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e2cf884-5d60-4e9b-9410-d8f424c2a1ee-proc\") pod \"perf-node-gather-daemonset-hqrfw\" (UID: \"0e2cf884-5d60-4e9b-9410-d8f424c2a1ee\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" Jul 27 17:13:45.368870 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.368735 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e2cf884-5d60-4e9b-9410-d8f424c2a1ee-lib-modules\") pod \"perf-node-gather-daemonset-hqrfw\" (UID: \"0e2cf884-5d60-4e9b-9410-d8f424c2a1ee\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" Jul 27 17:13:45.469683 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.469650 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e2cf884-5d60-4e9b-9410-d8f424c2a1ee-proc\") pod \"perf-node-gather-daemonset-hqrfw\" (UID: \"0e2cf884-5d60-4e9b-9410-d8f424c2a1ee\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" Jul 27 17:13:45.470133 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.469695 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e2cf884-5d60-4e9b-9410-d8f424c2a1ee-lib-modules\") pod \"perf-node-gather-daemonset-hqrfw\" (UID: \"0e2cf884-5d60-4e9b-9410-d8f424c2a1ee\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" Jul 27 17:13:45.470133 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.469734 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhqqq\" (UniqueName: \"kubernetes.io/projected/0e2cf884-5d60-4e9b-9410-d8f424c2a1ee-kube-api-access-nhqqq\") pod \"perf-node-gather-daemonset-hqrfw\" (UID: \"0e2cf884-5d60-4e9b-9410-d8f424c2a1ee\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" Jul 27 17:13:45.470133 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.469771 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e2cf884-5d60-4e9b-9410-d8f424c2a1ee-podres\") pod \"perf-node-gather-daemonset-hqrfw\" (UID: \"0e2cf884-5d60-4e9b-9410-d8f424c2a1ee\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" Jul 27 17:13:45.470133 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.469778 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e2cf884-5d60-4e9b-9410-d8f424c2a1ee-proc\") pod \"perf-node-gather-daemonset-hqrfw\" (UID: \"0e2cf884-5d60-4e9b-9410-d8f424c2a1ee\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" Jul 27 17:13:45.470133 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.469789 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e2cf884-5d60-4e9b-9410-d8f424c2a1ee-sys\") pod \"perf-node-gather-daemonset-hqrfw\" (UID: \"0e2cf884-5d60-4e9b-9410-d8f424c2a1ee\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" Jul 27 17:13:45.470133 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.469868 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e2cf884-5d60-4e9b-9410-d8f424c2a1ee-sys\") pod \"perf-node-gather-daemonset-hqrfw\" (UID: \"0e2cf884-5d60-4e9b-9410-d8f424c2a1ee\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" Jul 27 17:13:45.470133 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.469891 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e2cf884-5d60-4e9b-9410-d8f424c2a1ee-lib-modules\") pod \"perf-node-gather-daemonset-hqrfw\" (UID: \"0e2cf884-5d60-4e9b-9410-d8f424c2a1ee\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" Jul 27 17:13:45.470133 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.469968 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e2cf884-5d60-4e9b-9410-d8f424c2a1ee-podres\") pod \"perf-node-gather-daemonset-hqrfw\" (UID: \"0e2cf884-5d60-4e9b-9410-d8f424c2a1ee\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" Jul 27 17:13:45.481560 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.481532 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhqqq\" (UniqueName: \"kubernetes.io/projected/0e2cf884-5d60-4e9b-9410-d8f424c2a1ee-kube-api-access-nhqqq\") pod \"perf-node-gather-daemonset-hqrfw\" (UID: \"0e2cf884-5d60-4e9b-9410-d8f424c2a1ee\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" Jul 27 17:13:45.583464 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.583360 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" Jul 27 17:13:45.734797 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.734773 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw"] Jul 27 17:13:45.736785 ip-10-0-135-70 kubenswrapper[2550]: W0727 17:13:45.736755 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0e2cf884_5d60_4e9b_9410_d8f424c2a1ee.slice/crio-40afe712e705a6a8d82a8175e851d8e79dab65ff20596dd2b83318c2fc61b8ed WatchSource:0}: Error finding container 40afe712e705a6a8d82a8175e851d8e79dab65ff20596dd2b83318c2fc61b8ed: Status 404 returned error can't find the container with id 40afe712e705a6a8d82a8175e851d8e79dab65ff20596dd2b83318c2fc61b8ed Jul 27 17:13:45.962581 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.962474 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" event={"ID":"0e2cf884-5d60-4e9b-9410-d8f424c2a1ee","Type":"ContainerStarted","Data":"07f91cc768f486cc0bf5095e4013008431bed6191804aadebef0120105aded08"} Jul 27 17:13:45.962581 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.962539 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" event={"ID":"0e2cf884-5d60-4e9b-9410-d8f424c2a1ee","Type":"ContainerStarted","Data":"40afe712e705a6a8d82a8175e851d8e79dab65ff20596dd2b83318c2fc61b8ed"} Jul 27 17:13:45.962768 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.962693 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" Jul 27 17:13:45.982005 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:45.981954 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" podStartSLOduration=0.981937876 podStartE2EDuration="981.937876ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:13:45.98015466 +0000 UTC m=+3404.652689160" watchObservedRunningTime="2026-07-27 17:13:45.981937876 +0000 UTC m=+3404.654472429" Jul 27 17:13:46.013458 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:46.013427 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-j4h8x_2337956f-94b4-44b9-9404-7cc9de6e6b0f/networking-console-plugin/0.log" Jul 27 17:13:47.168383 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:47.168355 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-78dc97c9f7-d6mqz_19a2f7be-7dea-4225-83b0-3aea575091bc/console/0.log" Jul 27 17:13:47.223691 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:47.223663 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-l48kt_223f7f5e-f930-4bfc-958d-9ec54d0a27a9/download-server/0.log" Jul 27 17:13:49.150093 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:49.150057 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f7sqd_76c82e0f-0e1b-400d-8c8f-9f5804aa15ac/dns/0.log" Jul 27 17:13:49.186854 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:49.186811 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f7sqd_76c82e0f-0e1b-400d-8c8f-9f5804aa15ac/kube-rbac-proxy/0.log" Jul 27 17:13:49.387581 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:49.387552 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9fnfd_92a27900-e3d3-44c2-a7da-7c3ff58b7cfc/dns-node-resolver/0.log" Jul 27 17:13:50.260625 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:50.260590 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tnb6j_9b8bc6ae-9d34-41e4-8079-9149a29e6627/node-ca/0.log" Jul 27 17:13:51.369058 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:51.369028 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799czw46d_022c0a96-b89f-484b-a715-6b90cc6cc408/istio-proxy/0.log" Jul 27 17:13:51.773111 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:51.773078 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78dbfdf94f-9trzp_31084f38-f413-40bf-a8ac-e5eaaa47b33a/kube-auth-proxy/0.log" Jul 27 17:13:51.916169 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:51.916134 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-kx6r6_d6e2c3b5-1303-4a7c-b111-2b0b661ff60c/payload-pre-processing/0.log" Jul 27 17:13:51.979452 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:51.979422 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-hqrfw" Jul 27 17:13:52.888227 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:52.888175 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dfvc2_26b55f2f-e26a-4918-83b8-d49688d388fd/serve-healthcheck-canary/0.log" Jul 27 17:13:53.477991 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:53.477965 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-dbfs5_359a50cb-e66a-4a77-8ae8-33bc152ad58f/insights-operator/0.log" Jul 27 17:13:53.479283 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:53.479256 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-dbfs5_359a50cb-e66a-4a77-8ae8-33bc152ad58f/insights-operator/1.log" Jul 27 17:13:53.572604 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:53.572577 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-68vxh_ba81a68e-6c72-4447-ad53-a81f29cee2a9/kube-rbac-proxy/0.log" Jul 27 17:13:53.597391 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:53.597363 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-68vxh_ba81a68e-6c72-4447-ad53-a81f29cee2a9/exporter/0.log" Jul 27 17:13:53.618600 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:53.618559 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-68vxh_ba81a68e-6c72-4447-ad53-a81f29cee2a9/extractor/0.log" Jul 27 17:13:56.230636 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:56.230598 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-74lck_c6e5dc62-7fb0-41d6-89c5-b4d0dc541a88/manager/0.log" Jul 27 17:13:56.481394 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:56.481320 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799czw46d_022c0a96-b89f-484b-a715-6b90cc6cc408/istio-proxy/0.log" Jul 27 17:13:56.622625 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:56.622567 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-cpc94_172833a1-d9d0-41f5-86a8-cf227d9a7479/manager/0.log" Jul 27 17:13:56.842113 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:56.842083 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78dbfdf94f-9trzp_31084f38-f413-40bf-a8ac-e5eaaa47b33a/kube-auth-proxy/0.log" Jul 27 17:13:56.944523 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:56.944482 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-kx6r6_d6e2c3b5-1303-4a7c-b111-2b0b661ff60c/payload-pre-processing/0.log" Jul 27 17:13:57.163669 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:57.163575 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55548c94b5-9zdpp_2730b6a9-a99f-4a61-b69c-74d48ee48c74/manager/0.log" Jul 27 17:13:57.182487 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:57.182463 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55548c94b5-9zdpp_2730b6a9-a99f-4a61-b69c-74d48ee48c74/copy-manifests/0.log" Jul 27 17:13:58.377829 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:58.377786 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799czw46d_022c0a96-b89f-484b-a715-6b90cc6cc408/istio-proxy/0.log" Jul 27 17:13:58.703119 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:58.703032 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78dbfdf94f-9trzp_31084f38-f413-40bf-a8ac-e5eaaa47b33a/kube-auth-proxy/0.log" Jul 27 17:13:58.806792 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:13:58.806764 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-kx6r6_d6e2c3b5-1303-4a7c-b111-2b0b661ff60c/payload-pre-processing/0.log" Jul 27 17:14:02.554353 ip-10-0-135-70 kubenswrapper[2550]: I0727 17:14:02.554321 2550 scope.go:117] "RemoveContainer" containerID="50a8a826f8cec5aca449b3dd31fd4af43b9ada022d5b8bccca63b483270f1fc7"