Jul 25 01:10:57.441038 ip-10-0-135-43 systemd[1]: Starting Kubernetes Kubelet... Jul 25 01:10:57.871066 ip-10-0-135-43 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 25 01:10:57.871066 ip-10-0-135-43 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 25 01:10:57.871066 ip-10-0-135-43 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 25 01:10:57.871066 ip-10-0-135-43 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 25 01:10:57.871066 ip-10-0-135-43 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 25 01:10:57.871968 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.871890 2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 25 01:10:57.876628 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876613 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 25 01:10:57.876628 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876629 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 25 01:10:57.876691 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876633 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 25 01:10:57.876691 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876636 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 25 01:10:57.876691 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876640 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 25 01:10:57.876691 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876643 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 25 01:10:57.876691 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876646 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 25 01:10:57.876691 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876649 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 25 01:10:57.876691 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876652 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 25 01:10:57.876691 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876656 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 25 01:10:57.876691 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876660 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 25 01:10:57.876691 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876663 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 25 01:10:57.876691 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876666 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 25 01:10:57.876691 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876668 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 25 01:10:57.876691 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876671 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 25 01:10:57.876691 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876673 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 25 01:10:57.876691 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876676 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 25 01:10:57.876691 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876679 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 25 01:10:57.876691 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876681 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 25 01:10:57.876691 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876683 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 25 01:10:57.876691 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876689 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876692 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876695 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876698 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876701 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876704 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876707 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876709 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876712 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876715 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876717 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876720 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876722 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876725 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876727 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876730 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876732 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876734 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876737 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876739 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 25 01:10:57.877149 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876742 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876744 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876747 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876783 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876788 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876791 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876794 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876797 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876800 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876802 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876805 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876807 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876810 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876812 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876816 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876818 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876821 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876824 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876826 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876829 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 25 01:10:57.877627 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876831 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 25 01:10:57.878125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876834 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 25 01:10:57.878125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876836 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 25 01:10:57.878125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876839 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 25 01:10:57.878125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876841 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 25 01:10:57.878125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876844 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 25 01:10:57.878125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876846 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 25 01:10:57.878125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876849 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 25 01:10:57.878125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876851 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 25 01:10:57.878125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876854 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 25 01:10:57.878125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876856 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 25 01:10:57.878125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876859 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 25 01:10:57.878125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876862 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 25 01:10:57.878125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876864 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 25 01:10:57.878125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876867 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 25 01:10:57.878125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876869 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 25 01:10:57.878125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876871 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 25 01:10:57.878125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876874 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 25 01:10:57.878125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876876 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 25 01:10:57.878125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876880 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 25 01:10:57.878573 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876883 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 25 01:10:57.878573 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876886 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 25 01:10:57.878573 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876888 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 25 01:10:57.878573 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876891 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 25 01:10:57.878573 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876894 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 25 01:10:57.878573 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876898 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 25 01:10:57.878573 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876901 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 25 01:10:57.878573 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876903 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 25 01:10:57.878573 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876906 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 25 01:10:57.878573 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876908 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 25 01:10:57.878573 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876911 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 25 01:10:57.878573 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876913 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 25 01:10:57.878573 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876915 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 25 01:10:57.878573 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876918 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 25 01:10:57.878573 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876921 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 25 01:10:57.878573 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876923 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 25 01:10:57.878573 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876925 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 25 01:10:57.878573 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876928 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 25 01:10:57.878573 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.876930 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 25 01:10:57.879033 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878843 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 25 01:10:57.879033 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878851 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 25 01:10:57.879033 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878855 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 25 01:10:57.879033 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878858 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 25 01:10:57.879033 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878861 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 25 01:10:57.879033 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878866 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 25 01:10:57.879033 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878870 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 25 01:10:57.879033 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878873 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 25 01:10:57.879033 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878875 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 25 01:10:57.879033 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878878 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 25 01:10:57.879033 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878881 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 25 01:10:57.879033 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878883 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 25 01:10:57.879033 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878886 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 25 01:10:57.879033 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878888 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 25 01:10:57.879033 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878891 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 25 01:10:57.879033 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878894 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 25 01:10:57.879033 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878896 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 25 01:10:57.879033 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878899 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 25 01:10:57.879033 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878901 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 25 01:10:57.879501 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878904 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 25 01:10:57.879501 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878906 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 25 01:10:57.879501 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878909 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 25 01:10:57.879501 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878911 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 25 01:10:57.879501 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878914 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 25 01:10:57.879501 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878916 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 25 01:10:57.879501 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878918 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 25 01:10:57.879501 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878922 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 25 01:10:57.879501 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878926 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 25 01:10:57.879501 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878928 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 25 01:10:57.879501 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878931 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 25 01:10:57.879501 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878933 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 25 01:10:57.879501 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878936 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 25 01:10:57.879501 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878938 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 25 01:10:57.879501 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878942 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 25 01:10:57.879501 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878944 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 25 01:10:57.879501 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878947 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 25 01:10:57.879501 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878949 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 25 01:10:57.879501 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878952 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878955 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878958 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878960 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878964 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878966 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878969 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878971 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878973 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878976 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878978 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878981 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878983 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878986 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878988 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878991 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878993 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878995 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.878998 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879000 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 25 01:10:57.879978 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879003 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879005 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879008 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879010 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879013 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879015 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879017 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879020 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879025 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879029 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879031 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879034 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879037 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879040 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879043 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879046 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879049 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879052 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879054 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879057 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 25 01:10:57.880449 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879060 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879063 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879065 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879068 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879071 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879073 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879076 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879078 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879081 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879083 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879085 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879088 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879090 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879093 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879095 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879098 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879100 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879103 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879105 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879108 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 25 01:10:57.880957 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879111 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879114 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879473 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879478 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879481 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879483 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879486 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879489 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879491 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879494 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879496 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879499 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879501 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879506 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879509 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879512 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879515 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879518 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879520 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879523 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 25 01:10:57.881436 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879526 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879529 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879532 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879534 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879537 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879539 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879541 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879544 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879546 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879549 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879552 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879554 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879557 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879561 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879563 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879566 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879568 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879570 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879573 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879576 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 25 01:10:57.881959 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879578 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 25 01:10:57.882446 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879581 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 25 01:10:57.882446 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879583 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 25 01:10:57.882446 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879585 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 25 01:10:57.882446 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879588 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 25 01:10:57.882446 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879590 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 25 01:10:57.882446 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879592 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 25 01:10:57.882446 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879595 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 25 01:10:57.882446 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879597 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 25 01:10:57.882446 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879599 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 25 01:10:57.882446 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879602 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 25 01:10:57.882446 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879604 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 25 01:10:57.882446 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879607 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 25 01:10:57.882446 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879609 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 25 01:10:57.882446 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879612 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 25 01:10:57.882446 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879615 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 25 01:10:57.882446 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879617 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 25 01:10:57.882446 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879619 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 25 01:10:57.882446 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879622 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 25 01:10:57.882446 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879624 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 25 01:10:57.882912 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879626 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 25 01:10:57.882912 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879629 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 25 01:10:57.882912 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879632 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 25 01:10:57.882912 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879634 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 25 01:10:57.882912 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879637 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 25 01:10:57.882912 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879640 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 25 01:10:57.882912 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879643 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 25 01:10:57.882912 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879645 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 25 01:10:57.882912 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879648 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 25 01:10:57.882912 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879650 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 25 01:10:57.882912 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879654 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 25 01:10:57.882912 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879657 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 25 01:10:57.882912 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879660 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 25 01:10:57.882912 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879663 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 25 01:10:57.882912 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879666 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 25 01:10:57.882912 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879668 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 25 01:10:57.882912 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879671 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 25 01:10:57.882912 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879673 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 25 01:10:57.882912 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879676 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879678 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879680 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879683 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879685 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879688 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879690 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879692 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879695 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879697 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879699 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879702 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879704 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879706 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879709 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879711 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879714 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879716 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879719 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879722 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 25 01:10:57.883393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879725 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879727 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.879729 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879825 2554 flags.go:64] FLAG: --address="0.0.0.0" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879839 2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879845 2554 flags.go:64] FLAG: --anonymous-auth="true" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879849 2554 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879854 2554 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879857 2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879860 2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879864 2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879867 2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879870 2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879873 2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879876 2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879879 2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879882 2554 flags.go:64] FLAG: --cgroup-root="" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879884 2554 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879887 2554 flags.go:64] FLAG: --client-ca-file="" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879890 2554 flags.go:64] FLAG: --cloud-provider="external" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879892 2554 flags.go:64] FLAG: --cluster-dns="[]" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879896 2554 flags.go:64] FLAG: --cluster-domain="" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879899 2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879902 2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 25 01:10:57.883875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879905 2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879908 2554 flags.go:64] FLAG: --container-log-max-files="5" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879911 2554 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879914 2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879917 2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879920 2554 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879923 2554 flags.go:64] FLAG: --contention-profiling="false" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879926 2554 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879928 2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879931 2554 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879934 2554 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879938 2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879941 2554 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879943 2554 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879946 2554 flags.go:64] FLAG: --enable-load-reader="false" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879949 2554 flags.go:64] FLAG: --enable-server="true" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879952 2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879956 2554 flags.go:64] FLAG: --event-burst="100" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879959 2554 flags.go:64] FLAG: --event-qps="50" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879961 2554 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879964 2554 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879967 2554 flags.go:64] FLAG: --eviction-hard="" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879970 2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879973 2554 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879975 2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 25 01:10:57.884533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879978 2554 flags.go:64] FLAG: --eviction-soft="" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879981 2554 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879984 2554 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879987 2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879990 2554 flags.go:64] FLAG: --experimental-mounter-path="" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879993 2554 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879996 2554 flags.go:64] FLAG: --fail-swap-on="true" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.879999 2554 flags.go:64] FLAG: --feature-gates="" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880004 2554 flags.go:64] FLAG: --file-check-frequency="20s" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880007 2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880010 2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880013 2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880016 2554 flags.go:64] FLAG: --healthz-port="10248" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880019 2554 flags.go:64] FLAG: --help="false" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880021 2554 flags.go:64] FLAG: --hostname-override="ip-10-0-135-43.ec2.internal" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880024 2554 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880027 2554 flags.go:64] FLAG: --http-check-frequency="20s" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880030 2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880033 2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880036 2554 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880039 2554 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880042 2554 flags.go:64] FLAG: --image-service-endpoint="" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880044 2554 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 25 01:10:57.885187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880047 2554 flags.go:64] FLAG: --kube-api-burst="100" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880050 2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880053 2554 flags.go:64] FLAG: --kube-api-qps="50" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880056 2554 flags.go:64] FLAG: --kube-reserved="" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880058 2554 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880061 2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880064 2554 flags.go:64] FLAG: --kubelet-cgroups="" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880066 2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880069 2554 flags.go:64] FLAG: --lock-file="" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880072 2554 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880074 2554 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880077 2554 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880082 2554 flags.go:64] FLAG: --log-json-split-stream="false" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880085 2554 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880087 2554 flags.go:64] FLAG: --log-text-split-stream="false" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880090 2554 flags.go:64] FLAG: --logging-format="text" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880093 2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880097 2554 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880100 2554 flags.go:64] FLAG: --manifest-url="" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880103 2554 flags.go:64] FLAG: --manifest-url-header="" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880106 2554 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880109 2554 flags.go:64] FLAG: --max-open-files="1000000" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880113 2554 flags.go:64] FLAG: --max-pods="110" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880116 2554 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880119 2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 25 01:10:57.885755 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880122 2554 flags.go:64] FLAG: --memory-manager-policy="None" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880124 2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880127 2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880130 2554 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880133 2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880139 2554 flags.go:64] FLAG: --node-status-max-images="50" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880142 2554 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880144 2554 flags.go:64] FLAG: --oom-score-adj="-999" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880147 2554 flags.go:64] FLAG: --pod-cidr="" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880150 2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880155 2554 flags.go:64] FLAG: --pod-manifest-path="" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880157 2554 flags.go:64] FLAG: --pod-max-pids="-1" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880160 2554 flags.go:64] FLAG: --pods-per-core="0" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880163 2554 flags.go:64] FLAG: --port="10250" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880166 2554 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880169 2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00672dd8d64bbea9b" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880172 2554 flags.go:64] FLAG: --qos-reserved="" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880174 2554 flags.go:64] FLAG: --read-only-port="10255" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880177 2554 flags.go:64] FLAG: --register-node="true" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880180 2554 flags.go:64] FLAG: --register-with-taints="" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880183 2554 flags.go:64] FLAG: --registry-burst="10" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880186 2554 flags.go:64] FLAG: --registry-qps="5" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880189 2554 flags.go:64] FLAG: --reserved-cpus="" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880192 2554 flags.go:64] FLAG: --reserved-memory="" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880195 2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 25 01:10:57.886371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880199 2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880203 2554 flags.go:64] FLAG: --rotate-certificates="false" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880205 2554 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880208 2554 flags.go:64] FLAG: --runonce="false" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880211 2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880213 2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880216 2554 flags.go:64] FLAG: --seccomp-default="false" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880219 2554 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880222 2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880225 2554 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880228 2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880231 2554 flags.go:64] FLAG: --storage-driver-password="root" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880234 2554 flags.go:64] FLAG: --storage-driver-secure="false" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880236 2554 flags.go:64] FLAG: --storage-driver-table="stats" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880239 2554 flags.go:64] FLAG: --storage-driver-user="root" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880242 2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880245 2554 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880248 2554 flags.go:64] FLAG: --system-cgroups="" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880251 2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880256 2554 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880258 2554 flags.go:64] FLAG: --tls-cert-file="" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880261 2554 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880265 2554 flags.go:64] FLAG: --tls-min-version="" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880268 2554 flags.go:64] FLAG: --tls-private-key-file="" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880271 2554 flags.go:64] FLAG: --topology-manager-policy="none" Jul 25 01:10:57.887411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880273 2554 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880276 2554 flags.go:64] FLAG: --topology-manager-scope="container" Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880280 2554 flags.go:64] FLAG: --v="2" Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880284 2554 flags.go:64] FLAG: --version="false" Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880288 2554 flags.go:64] FLAG: --vmodule="" Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880291 2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.880294 2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880403 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880408 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880411 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880414 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880417 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880419 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880422 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880425 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880428 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880430 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880434 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880437 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880440 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 25 01:10:57.888489 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880443 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880446 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880448 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880452 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880454 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880458 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880461 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880471 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880476 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880479 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880481 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880484 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880486 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880489 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880493 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880495 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880498 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880500 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880503 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880507 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 25 01:10:57.889415 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880509 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880512 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880514 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880518 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880521 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880524 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880527 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880529 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880532 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880534 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880537 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880539 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880542 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880544 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880547 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880549 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880552 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880554 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880557 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880560 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 25 01:10:57.890246 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880562 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880565 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880567 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880570 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880572 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880574 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880578 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880581 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880583 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880586 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880588 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880592 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880594 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880597 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880599 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880602 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880605 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880607 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880610 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880612 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 25 01:10:57.890873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880614 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880617 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880619 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880622 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880624 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880626 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880629 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880631 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880635 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880639 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880642 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880645 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880648 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880650 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880652 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880655 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880657 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880660 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880664 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880666 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 25 01:10:57.891698 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880668 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 25 01:10:57.892358 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880671 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 25 01:10:57.892358 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880673 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 25 01:10:57.892358 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880678 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 25 01:10:57.892358 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880680 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 25 01:10:57.892358 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.880683 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 25 01:10:57.892358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.881461 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 25 01:10:57.892358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.888653 2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 25 01:10:57.892358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.888671 2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 25 01:10:57.892358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.888689 2554 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 25 01:10:57.892358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.888702 2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 25 01:10:57.892358 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888806 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 25 01:10:57.892358 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888814 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 25 01:10:57.892358 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888819 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 25 01:10:57.892358 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888824 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 25 01:10:57.892358 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888828 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 25 01:10:57.892890 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888832 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 25 01:10:57.892890 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888837 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 25 01:10:57.892890 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888841 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 25 01:10:57.892890 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888845 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 25 01:10:57.892890 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888849 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 25 01:10:57.892890 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888856 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 25 01:10:57.892890 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888862 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 25 01:10:57.892890 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888867 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 25 01:10:57.892890 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888872 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 25 01:10:57.892890 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888877 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 25 01:10:57.892890 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888882 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 25 01:10:57.892890 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888886 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 25 01:10:57.892890 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888891 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 25 01:10:57.892890 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888896 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 25 01:10:57.892890 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888900 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 25 01:10:57.892890 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888904 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 25 01:10:57.892890 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888909 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 25 01:10:57.892890 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888913 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 25 01:10:57.892890 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888917 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888921 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888925 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888929 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888933 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888937 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888943 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888947 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888952 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888956 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888960 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888965 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888968 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888972 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888976 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888980 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888984 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888988 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888992 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.888997 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 25 01:10:57.893373 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889001 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889005 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889009 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889014 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889018 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889022 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889026 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889030 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889034 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889038 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889044 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889051 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889055 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889059 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889063 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889067 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889072 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889075 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889081 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889086 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 25 01:10:57.893984 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889091 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889095 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889099 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889103 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889108 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889112 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889116 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889122 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889128 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889133 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889137 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889142 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889146 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889150 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889154 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889158 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889162 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889167 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889171 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889176 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 25 01:10:57.894523 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889180 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 25 01:10:57.895072 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889184 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 25 01:10:57.895072 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889188 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 25 01:10:57.895072 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889192 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 25 01:10:57.895072 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889196 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 25 01:10:57.895072 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889200 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 25 01:10:57.895072 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889204 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 25 01:10:57.895072 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889208 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 25 01:10:57.895072 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889212 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 25 01:10:57.895072 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889217 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 25 01:10:57.895072 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889222 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 25 01:10:57.895072 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889226 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 25 01:10:57.895072 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889231 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 25 01:10:57.895072 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889235 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 25 01:10:57.895072 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889239 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 25 01:10:57.895072 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889243 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 25 01:10:57.895444 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.889251 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 25 01:10:57.895444 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889431 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 25 01:10:57.895444 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889440 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 25 01:10:57.895444 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889444 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 25 01:10:57.895444 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889449 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 25 01:10:57.895444 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889454 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 25 01:10:57.895444 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889459 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 25 01:10:57.895444 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889463 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 25 01:10:57.895444 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889468 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 25 01:10:57.895444 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889472 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 25 01:10:57.895444 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889477 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 25 01:10:57.895444 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889481 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 25 01:10:57.895444 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889485 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 25 01:10:57.895444 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889490 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 25 01:10:57.895444 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889494 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889498 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889502 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889506 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889510 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889516 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889520 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889524 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889528 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889533 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889537 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889541 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889545 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889549 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889553 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889558 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889563 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889567 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889571 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889575 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 25 01:10:57.895864 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889579 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889585 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889592 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889596 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889600 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889605 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889609 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889613 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889618 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889622 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889626 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889630 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889634 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889639 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889643 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889647 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889651 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889654 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889658 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889663 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 25 01:10:57.896356 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889667 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889671 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889675 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889680 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889684 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889688 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889692 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889696 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889702 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889706 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889710 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889714 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889718 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889722 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889727 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889731 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889735 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889739 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889742 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889747 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 25 01:10:57.897177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889750 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 25 01:10:57.897660 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889755 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 25 01:10:57.897660 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889759 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 25 01:10:57.897660 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889765 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 25 01:10:57.897660 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889790 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 25 01:10:57.897660 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889795 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 25 01:10:57.897660 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889800 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 25 01:10:57.897660 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889804 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 25 01:10:57.897660 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889808 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 25 01:10:57.897660 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889812 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 25 01:10:57.897660 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889815 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 25 01:10:57.897660 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889819 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 25 01:10:57.897660 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889824 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 25 01:10:57.897660 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889828 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 25 01:10:57.897660 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889832 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 25 01:10:57.897660 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889837 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 25 01:10:57.897660 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889843 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 25 01:10:57.897660 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889849 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 25 01:10:57.897660 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889853 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 25 01:10:57.897660 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889857 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 25 01:10:57.898180 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889861 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 25 01:10:57.898180 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889867 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 25 01:10:57.898180 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889871 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 25 01:10:57.898180 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889875 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 25 01:10:57.898180 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889879 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 25 01:10:57.898180 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:10:57.889883 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 25 01:10:57.898180 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.889891 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 25 01:10:57.898180 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.890042 2554 server.go:962] "Client rotation is on, will bootstrap in background" Jul 25 01:10:57.898180 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.892627 2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 25 01:10:57.898180 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.894156 2554 server.go:1019] "Starting client certificate rotation" Jul 25 01:10:57.898180 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.894254 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 25 01:10:57.898180 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.894290 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 25 01:10:57.898180 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.896957 2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 25 01:10:57.904176 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.904156 2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 25 01:10:57.916479 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.916118 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 25 01:10:57.919707 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.919690 2554 log.go:25] "Validated CRI v1 runtime API" Jul 25 01:10:57.922381 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.922368 2554 log.go:25] "Validated CRI v1 image API" Jul 25 01:10:57.923165 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.923150 2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 25 01:10:57.926008 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.925991 2554 fs.go:135] Filesystem UUIDs: map[252bcd3c-9283-4e7d-8a07-63465394a861:/dev/nvme0n1p4 30778ae6-019a-47b2-8d13-cd6fdc81a1d0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 25 01:10:57.926065 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.926008 2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 25 01:10:57.931801 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.931687 2554 manager.go:216] Machine: {Timestamp:2026-07-25 01:10:57.930305993 +0000 UTC m=+0.366819480 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101363 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2726a77bf08e80959e6e1d748dcade SystemUUID:ec2726a7-7bf0-8e80-959e-6e1d748dcade BootID:20f0649b-e570-4c79-b487-5c9557eb3fc5 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b9:a7:8b:7d:41 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b9:a7:8b:7d:41 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:8c:c8:19:82:ac Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 25 01:10:57.931801 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.931800 2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 25 01:10:57.931894 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.931876 2554 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 25 01:10:57.934026 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.934005 2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 25 01:10:57.934157 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.934027 2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-43.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 25 01:10:57.935322 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.935311 2554 topology_manager.go:138] "Creating topology manager with none policy" Jul 25 01:10:57.935355 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.935325 2554 container_manager_linux.go:309] "Creating device plugin manager" Jul 25 01:10:57.935355 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.935332 2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 25 01:10:57.935355 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.935347 2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 25 01:10:57.936215 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.936205 2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 25 01:10:57.938845 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.938834 2554 state_mem.go:36] "Initialized new in-memory state store" Jul 25 01:10:57.938950 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.938941 2554 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 25 01:10:57.938987 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.938970 2554 kubelet.go:486] "Attempting to sync node with API server" Jul 25 01:10:57.938987 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.938979 2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 25 01:10:57.939037 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.938993 2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 25 01:10:57.939037 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.939003 2554 kubelet.go:398] "Adding apiserver pod source" Jul 25 01:10:57.939037 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.939011 2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 25 01:10:57.939425 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.939409 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 25 01:10:57.939473 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.939437 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 25 01:10:57.941330 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.941314 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pfzk5" Jul 25 01:10:57.941757 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.941740 2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 25 01:10:57.943514 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.943497 2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 25 01:10:57.943592 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.943536 2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 25 01:10:57.945285 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.945268 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 25 01:10:57.945365 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.945292 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 25 01:10:57.945365 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.945301 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 25 01:10:57.945365 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.945309 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 25 01:10:57.945365 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.945321 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 25 01:10:57.945365 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.945330 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 25 01:10:57.945365 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.945342 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 25 01:10:57.945365 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.945350 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 25 01:10:57.945365 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.945361 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 25 01:10:57.945365 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.945369 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 25 01:10:57.945726 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.945383 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 25 01:10:57.945726 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.945391 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 25 01:10:57.945726 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.945420 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 25 01:10:57.945726 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.945429 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 25 01:10:57.947843 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.947822 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pfzk5" Jul 25 01:10:57.948963 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.948937 2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-43.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 25 01:10:57.949410 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.949393 2554 server.go:1268] "Started kubelet" Jul 25 01:10:57.949965 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:10:57.949932 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 25 01:10:57.950078 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.949954 2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 25 01:10:57.950380 ip-10-0-135-43 systemd[1]: Started Kubernetes Kubelet. Jul 25 01:10:57.950932 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:10:57.949935 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 25 01:10:57.951096 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.950861 2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 25 01:10:57.951156 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.951131 2554 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 25 01:10:57.952694 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.952673 2554 server.go:310] "Adding debug handlers to kubelet server" Jul 25 01:10:57.952918 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.952873 2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 25 01:10:57.956016 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:10:57.955996 2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 25 01:10:57.959801 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:10:57.958617 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-43.ec2.internal.18c56329abcbfdd5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-43.ec2.internal,UID:ip-10-0-135-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-43.ec2.internal,},FirstTimestamp:2026-07-25 01:10:57.949351381 +0000 UTC m=+0.385864870,LastTimestamp:2026-07-25 01:10:57.949351381 +0000 UTC m=+0.385864870,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-43.ec2.internal,}" Jul 25 01:10:57.959902 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.959837 2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 25 01:10:57.959902 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.959857 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 25 01:10:57.960003 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:10:57.959917 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found" Jul 25 01:10:57.960437 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.960418 2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 25 01:10:57.960520 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.960433 2554 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 25 01:10:57.960520 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.960452 2554 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 25 01:10:57.960595 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.960536 2554 reconstruct.go:97] "Volume reconstruction finished" Jul 25 01:10:57.960595 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.960544 2554 reconciler.go:29] "Reconciler: start to sync state" Jul 25 01:10:57.960688 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.960640 2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 25 01:10:57.960688 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.960654 2554 factory.go:55] Registering systemd factory Jul 25 01:10:57.960688 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.960663 2554 factory.go:223] Registration of the systemd container factory successfully Jul 25 01:10:57.960894 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.960875 2554 factory.go:153] Registering CRI-O factory Jul 25 01:10:57.960894 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.960895 2554 factory.go:223] Registration of the crio container factory successfully Jul 25 01:10:57.961017 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.960917 2554 factory.go:103] Registering Raw factory Jul 25 01:10:57.961017 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.960928 2554 manager.go:1196] Started watching for new ooms in manager Jul 25 01:10:57.961403 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.961392 2554 manager.go:349] Starting recovery of all containers Jul 25 01:10:57.971808 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.971655 2554 manager.go:354] Recovery completed Jul 25 01:10:57.972802 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.972765 2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 25 01:10:57.975601 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:10:57.975585 2554 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-43.ec2.internal\" not found" node="ip-10-0-135-43.ec2.internal" Jul 25 01:10:57.976839 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.976810 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 25 01:10:57.978554 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.978536 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientMemory" Jul 25 01:10:57.978613 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.978566 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 25 01:10:57.978613 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.978576 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientPID" Jul 25 01:10:57.979068 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.979054 2554 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 25 01:10:57.979068 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.979063 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 25 01:10:57.979068 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.979068 2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 25 01:10:57.979271 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.979083 2554 state_mem.go:36] "Initialized new in-memory state store" Jul 25 01:10:57.980214 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.980200 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientMemory" Jul 25 01:10:57.980287 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.980230 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 25 01:10:57.980287 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.980245 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientPID" Jul 25 01:10:57.981675 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.981663 2554 policy_none.go:49] "None policy: Start" Jul 25 01:10:57.982319 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.982307 2554 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 25 01:10:57.982354 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.982329 2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 25 01:10:57.984284 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:57.984271 2554 policy_none.go:47] "Start" Jul 25 01:10:58.021324 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.021309 2554 manager.go:337] "Starting Device Plugin manager" Jul 25 01:10:58.021412 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:10:58.021343 2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 25 01:10:58.021412 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.021353 2554 server.go:87] "Starting device plugin registration server" Jul 25 01:10:58.021565 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.021553 2554 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 25 01:10:58.021604 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.021568 2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 25 01:10:58.021664 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.021644 2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 25 01:10:58.021735 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.021718 2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 25 01:10:58.021735 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.021730 2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 25 01:10:58.022176 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:10:58.022159 2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 25 01:10:58.022240 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:10:58.022212 2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-43.ec2.internal\" not found" Jul 25 01:10:58.058220 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.058191 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 25 01:10:58.065462 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.059524 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 25 01:10:58.065462 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.059554 2554 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 25 01:10:58.065462 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.059571 2554 kubelet.go:2448] "Starting kubelet main sync loop" Jul 25 01:10:58.065462 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:10:58.059605 2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 25 01:10:58.065462 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.062249 2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 25 01:10:58.122503 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.122431 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 25 01:10:58.123324 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.123307 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientMemory" Jul 25 01:10:58.123408 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.123338 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 25 01:10:58.123408 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.123349 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientPID" Jul 25 01:10:58.123408 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.123370 2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-43.ec2.internal" Jul 25 01:10:58.131421 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.131405 2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-43.ec2.internal" Jul 25 01:10:58.131471 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:10:58.131428 2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-43.ec2.internal\": node \"ip-10-0-135-43.ec2.internal\" not found" Jul 25 01:10:58.145425 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:10:58.145404 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found" Jul 25 01:10:58.160657 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.160632 2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal"] Jul 25 01:10:58.160717 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.160690 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 25 01:10:58.162296 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.162276 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientMemory" Jul 25 01:10:58.162366 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.162311 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 25 01:10:58.162366 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.162321 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientPID" Jul 25 01:10:58.164820 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.164807 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 25 01:10:58.164874 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.164860 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal" Jul 25 01:10:58.164911 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.164891 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 25 01:10:58.165542 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.165528 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientMemory" Jul 25 01:10:58.165606 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.165554 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 25 01:10:58.165606 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.165528 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientMemory" Jul 25 01:10:58.165606 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.165588 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 25 01:10:58.165606 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.165601 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientPID" Jul 25 01:10:58.165757 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.165564 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientPID" Jul 25 01:10:58.167953 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.167939 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal" Jul 25 01:10:58.167997 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.167967 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 25 01:10:58.168649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.168634 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientMemory" Jul 25 01:10:58.168725 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.168660 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 25 01:10:58.168725 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.168674 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientPID" Jul 25 01:10:58.199397 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:10:58.199377 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-43.ec2.internal\" not found" node="ip-10-0-135-43.ec2.internal" Jul 25 01:10:58.203951 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:10:58.203936 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-43.ec2.internal\" not found" node="ip-10-0-135-43.ec2.internal" Jul 25 01:10:58.245465 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:10:58.245441 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found" Jul 25 01:10:58.261727 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.261706 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1bc6362a334464c48c83d833f37a3ca-config\") pod \"kube-apiserver-proxy-ip-10-0-135-43.ec2.internal\" (UID: \"a1bc6362a334464c48c83d833f37a3ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal" Jul 25 01:10:58.261826 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.261737 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b8fdbf3fd61f65533c9d5667ec910c4a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal\" (UID: \"b8fdbf3fd61f65533c9d5667ec910c4a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal" Jul 25 01:10:58.261826 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.261764 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8fdbf3fd61f65533c9d5667ec910c4a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal\" (UID: \"b8fdbf3fd61f65533c9d5667ec910c4a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal" Jul 25 01:10:58.345799 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:10:58.345767 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found" Jul 25 01:10:58.362060 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.362037 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8fdbf3fd61f65533c9d5667ec910c4a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal\" (UID: \"b8fdbf3fd61f65533c9d5667ec910c4a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal" Jul 25 01:10:58.362174 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.362064 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1bc6362a334464c48c83d833f37a3ca-config\") pod \"kube-apiserver-proxy-ip-10-0-135-43.ec2.internal\" (UID: \"a1bc6362a334464c48c83d833f37a3ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal" Jul 25 01:10:58.362174 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.362082 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b8fdbf3fd61f65533c9d5667ec910c4a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal\" (UID: \"b8fdbf3fd61f65533c9d5667ec910c4a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal" Jul 25 01:10:58.362174 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.362122 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b8fdbf3fd61f65533c9d5667ec910c4a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal\" (UID: \"b8fdbf3fd61f65533c9d5667ec910c4a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal" Jul 25 01:10:58.362174 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.362137 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8fdbf3fd61f65533c9d5667ec910c4a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal\" (UID: \"b8fdbf3fd61f65533c9d5667ec910c4a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal" Jul 25 01:10:58.362310 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.362171 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1bc6362a334464c48c83d833f37a3ca-config\") pod \"kube-apiserver-proxy-ip-10-0-135-43.ec2.internal\" (UID: \"a1bc6362a334464c48c83d833f37a3ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal" Jul 25 01:10:58.446491 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:10:58.446429 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found" Jul 25 01:10:58.501934 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.501905 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal" Jul 25 01:10:58.506899 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:10:58.506685 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal" Jul 25 01:10:58.546812 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:10:58.546786 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found" Jul 25 01:10:58.647301 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:10:58.647266 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found" Jul 25 01:11:00.062864 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:00.062840 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found" Jul 25 01:11:00.163337 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:00.163308 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found" Jul 25 01:11:00.208846 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:00.208820 2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 25 01:11:00.209259 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:00.208959 2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 25 01:11:00.209259 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:00.208971 2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 25 01:11:00.264234 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:00.264211 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found" Jul 25 01:11:00.271947 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:00.271915 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-24 01:05:59 +0000 UTC" deadline="2028-01-18 01:47:57.578330463 +0000 UTC" Jul 25 01:11:00.271947 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:00.271941 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13008h36m57.306392444s" Jul 25 01:11:00.314703 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:00.314662 2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 25 01:11:00.350310 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:00.350282 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8fdbf3fd61f65533c9d5667ec910c4a.slice/crio-eb1d81bba0700eb6e0d51ac79e2c0e8cd2176095325f48f05cf7fcd321b32c63 WatchSource:0}: Error finding container eb1d81bba0700eb6e0d51ac79e2c0e8cd2176095325f48f05cf7fcd321b32c63: Status 404 returned error can't find the container with id eb1d81bba0700eb6e0d51ac79e2c0e8cd2176095325f48f05cf7fcd321b32c63 Jul 25 01:11:00.350736 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:00.350717 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1bc6362a334464c48c83d833f37a3ca.slice/crio-88413b3b5e41053b5be50e262ca86f9f041d4a25eee7dd640d96e5a05e0cb6d5 WatchSource:0}: Error finding container 88413b3b5e41053b5be50e262ca86f9f041d4a25eee7dd640d96e5a05e0cb6d5: Status 404 returned error can't find the container with id 88413b3b5e41053b5be50e262ca86f9f041d4a25eee7dd640d96e5a05e0cb6d5 Jul 25 01:11:00.355835 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:00.355820 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 25 01:11:00.364494 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:00.364478 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found" Jul 25 01:11:00.371353 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:00.371337 2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 25 01:11:00.375980 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:00.375958 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal" Jul 25 01:11:00.378501 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:00.378456 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal" event={"ID":"a1bc6362a334464c48c83d833f37a3ca","Type":"ContainerStarted","Data":"88413b3b5e41053b5be50e262ca86f9f041d4a25eee7dd640d96e5a05e0cb6d5"} Jul 25 01:11:00.379528 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:00.379511 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal" event={"ID":"b8fdbf3fd61f65533c9d5667ec910c4a","Type":"ContainerStarted","Data":"eb1d81bba0700eb6e0d51ac79e2c0e8cd2176095325f48f05cf7fcd321b32c63"} Jul 25 01:11:00.386001 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:00.385985 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 25 01:11:00.387193 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:00.387181 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal" Jul 25 01:11:00.394465 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:00.394449 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 25 01:11:01.178151 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.178120 2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 25 01:11:01.255196 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.255165 2554 apiserver.go:52] "Watching apiserver" Jul 25 01:11:01.266187 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.266160 2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 25 01:11:01.266538 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.266516 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-jm7hh","kube-system/konnectivity-agent-fkgvk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj","openshift-cluster-node-tuning-operator/tuned-8lkx2","openshift-image-registry/node-ca-tb5g6","openshift-multus/multus-additional-cni-plugins-xhsbs","openshift-network-diagnostics/network-check-target-2498t","openshift-network-operator/iptables-alerter-5jxlb","kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal","openshift-dns/node-resolver-9sn77","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal","openshift-multus/multus-cgj8z","openshift-multus/network-metrics-daemon-xtmst","openshift-ovn-kubernetes/ovnkube-node-lph8b"] Jul 25 01:11:01.269084 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.269061 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:01.269187 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:01.269153 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2498t" podUID="6727d4bf-1e34-43aa-8766-cc4e224468b3" Jul 25 01:11:01.271199 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.271176 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:01.271304 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:01.271261 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtmst" podUID="55676d4a-8fb7-483b-b880-232cc2d10465" Jul 25 01:11:01.273375 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.273353 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.275116 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.275094 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 25 01:11:01.275490 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.275468 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 25 01:11:01.275732 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.275717 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 25 01:11:01.275820 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.275787 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gj7pg\"" Jul 25 01:11:01.275905 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.275892 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 25 01:11:01.276040 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.276025 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.277632 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.277611 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tb5g6" Jul 25 01:11:01.278404 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.278380 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 25 01:11:01.278404 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.278381 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kfkrs\"" Jul 25 01:11:01.278564 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.278433 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 25 01:11:01.279787 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.279769 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 25 01:11:01.279897 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.279787 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 25 01:11:01.279960 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.279940 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k9mw9\"" Jul 25 01:11:01.280016 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.279980 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 25 01:11:01.280558 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.280539 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.282557 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.282355 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 25 01:11:01.282557 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.282397 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 25 01:11:01.282557 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.282410 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8k95n\"" Jul 25 01:11:01.282557 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.282358 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 25 01:11:01.282557 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.282414 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 25 01:11:01.282870 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.282706 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 25 01:11:01.283023 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.283003 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.283950 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.283932 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dfd8d67b-61d5-4a13-b319-3a64776c0a5b-serviceca\") pod \"node-ca-tb5g6\" (UID: \"dfd8d67b-61d5-4a13-b319-3a64776c0a5b\") " pod="openshift-image-registry/node-ca-tb5g6" Jul 25 01:11:01.284034 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.283960 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l9kr\" (UniqueName: \"kubernetes.io/projected/dfd8d67b-61d5-4a13-b319-3a64776c0a5b-kube-api-access-5l9kr\") pod \"node-ca-tb5g6\" (UID: \"dfd8d67b-61d5-4a13-b319-3a64776c0a5b\") " pod="openshift-image-registry/node-ca-tb5g6" Jul 25 01:11:01.284034 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.283975 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs\") pod \"network-metrics-daemon-xtmst\" (UID: \"55676d4a-8fb7-483b-b880-232cc2d10465\") " pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:01.284034 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.283990 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fda1253f-1d94-4dc3-a2ce-5694538260dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.284034 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284014 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fda1253f-1d94-4dc3-a2ce-5694538260dc-sys-fs\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.284271 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284053 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-sysctl-d\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.284271 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284092 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-systemd\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.284271 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284116 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-tuned\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.284271 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284132 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-run\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.284271 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284147 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-sys\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.284271 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284168 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-lib-modules\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.284271 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284202 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-host\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.284271 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284239 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fda1253f-1d94-4dc3-a2ce-5694538260dc-device-dir\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.284570 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284277 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr4sx\" (UniqueName: \"kubernetes.io/projected/fda1253f-1d94-4dc3-a2ce-5694538260dc-kube-api-access-vr4sx\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.284570 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284313 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-modprobe-d\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.284570 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284345 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-var-lib-kubelet\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.284570 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284374 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dfd8d67b-61d5-4a13-b319-3a64776c0a5b-host\") pod \"node-ca-tb5g6\" (UID: \"dfd8d67b-61d5-4a13-b319-3a64776c0a5b\") " pod="openshift-image-registry/node-ca-tb5g6" Jul 25 01:11:01.284570 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284397 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6464s\" (UniqueName: \"kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s\") pod \"network-check-target-2498t\" (UID: \"6727d4bf-1e34-43aa-8766-cc4e224468b3\") " pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:01.284570 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284420 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m9c7\" (UniqueName: \"kubernetes.io/projected/55676d4a-8fb7-483b-b880-232cc2d10465-kube-api-access-9m9c7\") pod \"network-metrics-daemon-xtmst\" (UID: \"55676d4a-8fb7-483b-b880-232cc2d10465\") " pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:01.284570 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284437 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fda1253f-1d94-4dc3-a2ce-5694538260dc-socket-dir\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.284570 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284451 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fda1253f-1d94-4dc3-a2ce-5694538260dc-registration-dir\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.284570 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284471 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhgr8\" (UniqueName: \"kubernetes.io/projected/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-kube-api-access-dhgr8\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.284570 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284521 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-sysconfig\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.284570 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284555 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-sysctl-conf\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.284570 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284569 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-tmp\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.284994 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284583 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fda1253f-1d94-4dc3-a2ce-5694538260dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.284994 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.284597 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-kubernetes\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.285094 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.285057 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 25 01:11:01.285234 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.285213 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 25 01:11:01.285321 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.285259 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 25 01:11:01.285321 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.285271 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sw9bt\"" Jul 25 01:11:01.285321 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.285213 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 25 01:11:01.285321 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.285284 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 25 01:11:01.285496 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.285271 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 25 01:11:01.285946 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.285917 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 25 01:11:01.286098 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.286083 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:01.286160 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:01.286143 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm7hh" podUID="62f24f02-b14f-4633-a4e0-bed308f33ada" Jul 25 01:11:01.290464 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.290441 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fkgvk" Jul 25 01:11:01.292339 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.292320 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 25 01:11:01.292433 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.292344 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 25 01:11:01.292433 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.292344 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9kw9r\"" Jul 25 01:11:01.292825 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.292788 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9sn77" Jul 25 01:11:01.294502 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.294483 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 25 01:11:01.294614 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.294600 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 25 01:11:01.294682 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.294655 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4msrs\"" Jul 25 01:11:01.295284 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.295267 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5jxlb" Jul 25 01:11:01.297050 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.297031 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 25 01:11:01.297144 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.297031 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 25 01:11:01.297362 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.297343 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5wvsp\"" Jul 25 01:11:01.297448 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.297363 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 25 01:11:01.297640 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.297620 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.299447 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.299428 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 25 01:11:01.299626 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.299608 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cslc9\"" Jul 25 01:11:01.304593 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.304574 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rshnd" Jul 25 01:11:01.310280 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.310266 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rshnd" Jul 25 01:11:01.379370 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.379336 2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 25 01:11:01.385691 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.385671 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62f24f02-b14f-4633-a4e0-bed308f33ada-dbus\") pod \"global-pull-secret-syncer-jm7hh\" (UID: \"62f24f02-b14f-4633-a4e0-bed308f33ada\") " pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:01.385795 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.385697 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/becef38c-52d9-4baa-bfe9-30f712afabdc-hosts-file\") pod \"node-resolver-9sn77\" (UID: \"becef38c-52d9-4baa-bfe9-30f712afabdc\") " pod="openshift-dns/node-resolver-9sn77" Jul 25 01:11:01.385795 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.385715 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk2kd\" (UniqueName: \"kubernetes.io/projected/becef38c-52d9-4baa-bfe9-30f712afabdc-kube-api-access-xk2kd\") pod \"node-resolver-9sn77\" (UID: \"becef38c-52d9-4baa-bfe9-30f712afabdc\") " pod="openshift-dns/node-resolver-9sn77" Jul 25 01:11:01.385795 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.385734 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-kubernetes\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.385795 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.385751 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs\") pod \"network-metrics-daemon-xtmst\" (UID: \"55676d4a-8fb7-483b-b880-232cc2d10465\") " pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:01.385795 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.385772 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fda1253f-1d94-4dc3-a2ce-5694538260dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.386071 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.385813 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fda1253f-1d94-4dc3-a2ce-5694538260dc-sys-fs\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.386071 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.385837 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-hostroot\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.386071 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.385858 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/684c7548-f719-4ab6-a653-3e95e6511d2e-multus-daemon-config\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.386071 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.385872 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-kubernetes\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.386071 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.385885 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fda1253f-1d94-4dc3-a2ce-5694538260dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.386071 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.385878 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-run-openvswitch\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.386071 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:01.385911 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 25 01:11:01.386071 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.385949 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fda1253f-1d94-4dc3-a2ce-5694538260dc-sys-fs\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.386071 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.385992 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aed04098-6863-4391-90f3-67c8279b4afe-ovn-node-metrics-cert\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.386071 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:01.386049 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs podName:55676d4a-8fb7-483b-b880-232cc2d10465 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:01.886019567 +0000 UTC m=+3.007505398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs") pod "network-metrics-daemon-xtmst" (UID: "55676d4a-8fb7-483b-b880-232cc2d10465") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 25 01:11:01.386520 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386089 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-tuned\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.386520 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386118 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-cnibin\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.386520 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386160 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxmbr\" (UniqueName: \"kubernetes.io/projected/684c7548-f719-4ab6-a653-3e95e6511d2e-kube-api-access-sxmbr\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.386520 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386192 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-run-netns\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.386520 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386217 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-run-ovn-kubernetes\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.386520 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386246 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/145204ec-0354-45f7-846e-95e3e5580c70-iptables-alerter-script\") pod \"iptables-alerter-5jxlb\" (UID: \"145204ec-0354-45f7-846e-95e3e5580c70\") " pod="openshift-network-operator/iptables-alerter-5jxlb" Jul 25 01:11:01.386520 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386268 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-lib-modules\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.386520 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386284 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr4sx\" (UniqueName: \"kubernetes.io/projected/fda1253f-1d94-4dc3-a2ce-5694538260dc-kube-api-access-vr4sx\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.386520 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386317 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-multus-cni-dir\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.386520 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386333 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-system-cni-dir\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.386520 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386347 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aed04098-6863-4391-90f3-67c8279b4afe-ovnkube-script-lib\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.386520 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386362 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/becef38c-52d9-4baa-bfe9-30f712afabdc-tmp-dir\") pod \"node-resolver-9sn77\" (UID: \"becef38c-52d9-4baa-bfe9-30f712afabdc\") " pod="openshift-dns/node-resolver-9sn77" Jul 25 01:11:01.386520 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386388 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dfd8d67b-61d5-4a13-b319-3a64776c0a5b-host\") pod \"node-ca-tb5g6\" (UID: \"dfd8d67b-61d5-4a13-b319-3a64776c0a5b\") " pod="openshift-image-registry/node-ca-tb5g6" Jul 25 01:11:01.386520 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386405 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6464s\" (UniqueName: \"kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s\") pod \"network-check-target-2498t\" (UID: \"6727d4bf-1e34-43aa-8766-cc4e224468b3\") " pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:01.386520 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386420 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-host-run-netns\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.386520 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386406 2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 25 01:11:01.386520 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386437 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-multus-conf-dir\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.387248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386453 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ndjt\" (UniqueName: \"kubernetes.io/projected/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-kube-api-access-4ndjt\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.387248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386497 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-var-lib-openvswitch\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.387248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386519 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-etc-openvswitch\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.387248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386528 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-lib-modules\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.387248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386546 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/145204ec-0354-45f7-846e-95e3e5580c70-host-slash\") pod \"iptables-alerter-5jxlb\" (UID: \"145204ec-0354-45f7-846e-95e3e5580c70\") " pod="openshift-network-operator/iptables-alerter-5jxlb" Jul 25 01:11:01.387248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386563 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fda1253f-1d94-4dc3-a2ce-5694538260dc-registration-dir\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.387248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386577 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-host-run-k8s-cni-cncf-io\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.387248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386594 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.387248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386624 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrqxz\" (UniqueName: \"kubernetes.io/projected/145204ec-0354-45f7-846e-95e3e5580c70-kube-api-access-lrqxz\") pod \"iptables-alerter-5jxlb\" (UID: \"145204ec-0354-45f7-846e-95e3e5580c70\") " pod="openshift-network-operator/iptables-alerter-5jxlb" Jul 25 01:11:01.387248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386662 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-host-var-lib-cni-multus\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.387248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386677 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-run-ovn\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.387248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386691 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttq4m\" (UniqueName: \"kubernetes.io/projected/aed04098-6863-4391-90f3-67c8279b4afe-kube-api-access-ttq4m\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.387248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386707 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fda1253f-1d94-4dc3-a2ce-5694538260dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.387248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386725 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-cni-bin\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.387248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386745 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-cni-netd\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.387248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386767 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-host-var-lib-cni-bin\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.387937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386786 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fda1253f-1d94-4dc3-a2ce-5694538260dc-registration-dir\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.387937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386815 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-kubelet\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.387937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.386934 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dfd8d67b-61d5-4a13-b319-3a64776c0a5b-host\") pod \"node-ca-tb5g6\" (UID: \"dfd8d67b-61d5-4a13-b319-3a64776c0a5b\") " pod="openshift-image-registry/node-ca-tb5g6" Jul 25 01:11:01.387937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387019 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-run-systemd\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.387937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387037 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fda1253f-1d94-4dc3-a2ce-5694538260dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.387937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387053 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dfd8d67b-61d5-4a13-b319-3a64776c0a5b-serviceca\") pod \"node-ca-tb5g6\" (UID: \"dfd8d67b-61d5-4a13-b319-3a64776c0a5b\") " pod="openshift-image-registry/node-ca-tb5g6" Jul 25 01:11:01.387937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387115 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l9kr\" (UniqueName: \"kubernetes.io/projected/dfd8d67b-61d5-4a13-b319-3a64776c0a5b-kube-api-access-5l9kr\") pod \"node-ca-tb5g6\" (UID: \"dfd8d67b-61d5-4a13-b319-3a64776c0a5b\") " pod="openshift-image-registry/node-ca-tb5g6" Jul 25 01:11:01.387937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387228 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-system-cni-dir\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.387937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387259 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret\") pod \"global-pull-secret-syncer-jm7hh\" (UID: \"62f24f02-b14f-4633-a4e0-bed308f33ada\") " pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:01.387937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387300 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6-agent-certs\") pod \"konnectivity-agent-fkgvk\" (UID: \"ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6\") " pod="kube-system/konnectivity-agent-fkgvk" Jul 25 01:11:01.387937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387324 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.387937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387363 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-sysctl-d\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.387937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387400 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-systemd\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.387937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387442 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-os-release\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.387937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387477 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-sysctl-d\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.387937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387503 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-etc-kubernetes\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.387937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387536 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-systemd\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.388649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387550 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dfd8d67b-61d5-4a13-b319-3a64776c0a5b-serviceca\") pod \"node-ca-tb5g6\" (UID: \"dfd8d67b-61d5-4a13-b319-3a64776c0a5b\") " pod="openshift-image-registry/node-ca-tb5g6" Jul 25 01:11:01.388649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387576 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6-konnectivity-ca\") pod \"konnectivity-agent-fkgvk\" (UID: \"ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6\") " pod="kube-system/konnectivity-agent-fkgvk" Jul 25 01:11:01.388649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387614 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-run\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.388649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387676 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-sys\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.388649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387698 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-run\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.388649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387742 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-sys\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.388649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387765 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-host\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.388649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387783 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fda1253f-1d94-4dc3-a2ce-5694538260dc-device-dir\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.388649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387816 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-multus-socket-dir-parent\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.388649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387841 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-host-var-lib-kubelet\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.388649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387843 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-host\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.388649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387861 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fda1253f-1d94-4dc3-a2ce-5694538260dc-device-dir\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.388649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387867 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aed04098-6863-4391-90f3-67c8279b4afe-ovnkube-config\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.388649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387899 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62f24f02-b14f-4633-a4e0-bed308f33ada-kubelet-config\") pod \"global-pull-secret-syncer-jm7hh\" (UID: \"62f24f02-b14f-4633-a4e0-bed308f33ada\") " pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:01.388649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387932 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-modprobe-d\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.388649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387951 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-var-lib-kubelet\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.388649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.387992 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-var-lib-kubelet\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.389358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388010 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aed04098-6863-4391-90f3-67c8279b4afe-env-overrides\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.389358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388042 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-modprobe-d\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.389358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388041 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m9c7\" (UniqueName: \"kubernetes.io/projected/55676d4a-8fb7-483b-b880-232cc2d10465-kube-api-access-9m9c7\") pod \"network-metrics-daemon-xtmst\" (UID: \"55676d4a-8fb7-483b-b880-232cc2d10465\") " pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:01.389358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388071 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fda1253f-1d94-4dc3-a2ce-5694538260dc-socket-dir\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.389358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388100 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-host-run-multus-certs\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.389358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388118 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.389358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388134 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhgr8\" (UniqueName: \"kubernetes.io/projected/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-kube-api-access-dhgr8\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.389358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388148 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-os-release\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.389358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388165 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-slash\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.389358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388166 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fda1253f-1d94-4dc3-a2ce-5694538260dc-socket-dir\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.389358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388198 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-sysconfig\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.389358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388260 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-sysconfig\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.389358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388280 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-sysctl-conf\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.389358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388307 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-tmp\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.389358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388331 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/684c7548-f719-4ab6-a653-3e95e6511d2e-cni-binary-copy\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.389358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388351 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.389358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388370 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-log-socket\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.390114 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388388 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-cnibin\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.390114 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388414 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-sysctl-conf\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.390114 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388417 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.390114 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388444 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-systemd-units\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.390114 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.388461 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-node-log\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.390114 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.389943 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-etc-tuned\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.390524 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.390507 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-tmp\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.395302 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:01.395274 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 25 01:11:01.395302 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:01.395299 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 25 01:11:01.395477 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:01.395312 2554 projected.go:196] Error preparing data for projected volume kube-api-access-6464s for pod openshift-network-diagnostics/network-check-target-2498t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 25 01:11:01.395477 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:01.395424 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s podName:6727d4bf-1e34-43aa-8766-cc4e224468b3 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:01.895404731 +0000 UTC m=+3.016890563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6464s" (UniqueName: "kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s") pod "network-check-target-2498t" (UID: "6727d4bf-1e34-43aa-8766-cc4e224468b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 25 01:11:01.398481 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.398463 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m9c7\" (UniqueName: \"kubernetes.io/projected/55676d4a-8fb7-483b-b880-232cc2d10465-kube-api-access-9m9c7\") pod \"network-metrics-daemon-xtmst\" (UID: \"55676d4a-8fb7-483b-b880-232cc2d10465\") " pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:01.398853 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.398831 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l9kr\" (UniqueName: \"kubernetes.io/projected/dfd8d67b-61d5-4a13-b319-3a64776c0a5b-kube-api-access-5l9kr\") pod \"node-ca-tb5g6\" (UID: \"dfd8d67b-61d5-4a13-b319-3a64776c0a5b\") " pod="openshift-image-registry/node-ca-tb5g6" Jul 25 01:11:01.398964 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.398866 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhgr8\" (UniqueName: \"kubernetes.io/projected/1bb6ecc9-98e0-411a-8a63-a5cce9773efb-kube-api-access-dhgr8\") pod \"tuned-8lkx2\" (UID: \"1bb6ecc9-98e0-411a-8a63-a5cce9773efb\") " pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.400236 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.400214 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr4sx\" (UniqueName: \"kubernetes.io/projected/fda1253f-1d94-4dc3-a2ce-5694538260dc-kube-api-access-vr4sx\") pod \"aws-ebs-csi-driver-node-lffgj\" (UID: \"fda1253f-1d94-4dc3-a2ce-5694538260dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.488949 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.488876 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-multus-socket-dir-parent\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.488949 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.488911 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-host-var-lib-kubelet\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.488949 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.488935 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aed04098-6863-4391-90f3-67c8279b4afe-ovnkube-config\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.489189 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.488958 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62f24f02-b14f-4633-a4e0-bed308f33ada-kubelet-config\") pod \"global-pull-secret-syncer-jm7hh\" (UID: \"62f24f02-b14f-4633-a4e0-bed308f33ada\") " pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:01.489189 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.488983 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aed04098-6863-4391-90f3-67c8279b4afe-env-overrides\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.489189 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.488998 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-host-var-lib-kubelet\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.489189 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489010 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-multus-socket-dir-parent\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.489189 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489042 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-host-run-multus-certs\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.489189 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489004 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-host-run-multus-certs\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.489189 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489066 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62f24f02-b14f-4633-a4e0-bed308f33ada-kubelet-config\") pod \"global-pull-secret-syncer-jm7hh\" (UID: \"62f24f02-b14f-4633-a4e0-bed308f33ada\") " pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:01.489189 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489086 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.489189 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489119 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-os-release\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.489189 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489142 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-slash\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.489189 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489170 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/684c7548-f719-4ab6-a653-3e95e6511d2e-cni-binary-copy\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.489189 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489195 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.489730 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489219 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-log-socket\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.489730 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489248 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-cnibin\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.489730 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489275 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.489730 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489268 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-os-release\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.489730 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489298 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-systemd-units\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.489730 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489322 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-node-log\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.489730 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489345 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62f24f02-b14f-4633-a4e0-bed308f33ada-dbus\") pod \"global-pull-secret-syncer-jm7hh\" (UID: \"62f24f02-b14f-4633-a4e0-bed308f33ada\") " pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:01.489730 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489396 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/becef38c-52d9-4baa-bfe9-30f712afabdc-hosts-file\") pod \"node-resolver-9sn77\" (UID: \"becef38c-52d9-4baa-bfe9-30f712afabdc\") " pod="openshift-dns/node-resolver-9sn77" Jul 25 01:11:01.489730 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489425 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.489730 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489418 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk2kd\" (UniqueName: \"kubernetes.io/projected/becef38c-52d9-4baa-bfe9-30f712afabdc-kube-api-access-xk2kd\") pod \"node-resolver-9sn77\" (UID: \"becef38c-52d9-4baa-bfe9-30f712afabdc\") " pod="openshift-dns/node-resolver-9sn77" Jul 25 01:11:01.489730 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489486 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-hostroot\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.489730 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489504 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/684c7548-f719-4ab6-a653-3e95e6511d2e-multus-daemon-config\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.489730 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489519 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-run-openvswitch\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.489730 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489533 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aed04098-6863-4391-90f3-67c8279b4afe-ovn-node-metrics-cert\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.489730 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489562 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-cnibin\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.489730 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489599 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxmbr\" (UniqueName: \"kubernetes.io/projected/684c7548-f719-4ab6-a653-3e95e6511d2e-kube-api-access-sxmbr\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.489730 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489641 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-run-netns\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.490525 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489641 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aed04098-6863-4391-90f3-67c8279b4afe-ovnkube-config\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.490525 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489659 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-run-ovn-kubernetes\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.490525 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489685 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/145204ec-0354-45f7-846e-95e3e5580c70-iptables-alerter-script\") pod \"iptables-alerter-5jxlb\" (UID: \"145204ec-0354-45f7-846e-95e3e5580c70\") " pod="openshift-network-operator/iptables-alerter-5jxlb" Jul 25 01:11:01.490525 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489702 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-slash\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.490525 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489713 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-multus-cni-dir\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.490525 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489725 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-cnibin\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.490525 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489745 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-hostroot\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.490525 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489752 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.490525 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489775 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-system-cni-dir\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.490525 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489827 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-multus-cni-dir\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.490525 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489835 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-cnibin\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.490525 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489839 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/684c7548-f719-4ab6-a653-3e95e6511d2e-cni-binary-copy\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.490525 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489867 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-system-cni-dir\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.490525 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489903 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-run-openvswitch\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.490525 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489528 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aed04098-6863-4391-90f3-67c8279b4afe-env-overrides\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.490525 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489828 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aed04098-6863-4391-90f3-67c8279b4afe-ovnkube-script-lib\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.490525 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489936 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-run-netns\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.490525 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489966 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/becef38c-52d9-4baa-bfe9-30f712afabdc-tmp-dir\") pod \"node-resolver-9sn77\" (UID: \"becef38c-52d9-4baa-bfe9-30f712afabdc\") " pod="openshift-dns/node-resolver-9sn77" Jul 25 01:11:01.491419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.489986 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-node-log\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.491419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490012 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-host-run-netns\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.491419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490030 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/684c7548-f719-4ab6-a653-3e95e6511d2e-multus-daemon-config\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.491419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490034 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-systemd-units\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.491419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490061 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-log-socket\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.491419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490066 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-multus-conf-dir\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.491419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490035 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-multus-conf-dir\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.491419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490103 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ndjt\" (UniqueName: \"kubernetes.io/projected/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-kube-api-access-4ndjt\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.491419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490152 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-var-lib-openvswitch\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.491419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490178 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-etc-openvswitch\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.491419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490205 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/145204ec-0354-45f7-846e-95e3e5580c70-host-slash\") pod \"iptables-alerter-5jxlb\" (UID: \"145204ec-0354-45f7-846e-95e3e5580c70\") " pod="openshift-network-operator/iptables-alerter-5jxlb" Jul 25 01:11:01.491419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490229 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/becef38c-52d9-4baa-bfe9-30f712afabdc-hosts-file\") pod \"node-resolver-9sn77\" (UID: \"becef38c-52d9-4baa-bfe9-30f712afabdc\") " pod="openshift-dns/node-resolver-9sn77" Jul 25 01:11:01.491419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490234 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.491419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490231 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-host-run-k8s-cni-cncf-io\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.491419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490264 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-run-ovn-kubernetes\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.491419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490268 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-host-run-k8s-cni-cncf-io\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.491419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490284 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.492020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490179 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62f24f02-b14f-4633-a4e0-bed308f33ada-dbus\") pod \"global-pull-secret-syncer-jm7hh\" (UID: \"62f24f02-b14f-4633-a4e0-bed308f33ada\") " pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:01.492020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490314 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrqxz\" (UniqueName: \"kubernetes.io/projected/145204ec-0354-45f7-846e-95e3e5580c70-kube-api-access-lrqxz\") pod \"iptables-alerter-5jxlb\" (UID: \"145204ec-0354-45f7-846e-95e3e5580c70\") " pod="openshift-network-operator/iptables-alerter-5jxlb" Jul 25 01:11:01.492020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490334 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/145204ec-0354-45f7-846e-95e3e5580c70-host-slash\") pod \"iptables-alerter-5jxlb\" (UID: \"145204ec-0354-45f7-846e-95e3e5580c70\") " pod="openshift-network-operator/iptables-alerter-5jxlb" Jul 25 01:11:01.492020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490340 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-host-var-lib-cni-multus\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.492020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490365 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-run-ovn\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.492020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490389 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttq4m\" (UniqueName: \"kubernetes.io/projected/aed04098-6863-4391-90f3-67c8279b4afe-kube-api-access-ttq4m\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.492020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490416 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-cni-bin\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.492020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490437 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-cni-netd\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.492020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490446 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/145204ec-0354-45f7-846e-95e3e5580c70-iptables-alerter-script\") pod \"iptables-alerter-5jxlb\" (UID: \"145204ec-0354-45f7-846e-95e3e5580c70\") " pod="openshift-network-operator/iptables-alerter-5jxlb" Jul 25 01:11:01.492020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490451 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-host-var-lib-cni-bin\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.492020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490481 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-host-var-lib-cni-bin\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.492020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490490 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-kubelet\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.492020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490512 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-run-systemd\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.492020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490506 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-run-ovn\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.492020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490529 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-system-cni-dir\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.492020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490530 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-kubelet\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.492020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490315 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-etc-openvswitch\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.492550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490546 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret\") pod \"global-pull-secret-syncer-jm7hh\" (UID: \"62f24f02-b14f-4633-a4e0-bed308f33ada\") " pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:01.492550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490575 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-host-var-lib-cni-multus\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.492550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490583 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-host-run-netns\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.492550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490597 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-system-cni-dir\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.492550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490631 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.492550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490617 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6-agent-certs\") pod \"konnectivity-agent-fkgvk\" (UID: \"ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6\") " pod="kube-system/konnectivity-agent-fkgvk" Jul 25 01:11:01.492550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490665 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.492550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490691 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-os-release\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.492550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490694 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-cni-netd\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.492550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490636 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-run-systemd\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.492550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490738 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-etc-kubernetes\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.492550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490764 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6-konnectivity-ca\") pod \"konnectivity-agent-fkgvk\" (UID: \"ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6\") " pod="kube-system/konnectivity-agent-fkgvk" Jul 25 01:11:01.492550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490666 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-host-cni-bin\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.492550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490870 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-os-release\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.492550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490995 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aed04098-6863-4391-90f3-67c8279b4afe-var-lib-openvswitch\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.492550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490998 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/684c7548-f719-4ab6-a653-3e95e6511d2e-etc-kubernetes\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.492550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.491032 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/becef38c-52d9-4baa-bfe9-30f712afabdc-tmp-dir\") pod \"node-resolver-9sn77\" (UID: \"becef38c-52d9-4baa-bfe9-30f712afabdc\") " pod="openshift-dns/node-resolver-9sn77" Jul 25 01:11:01.492550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.490754 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aed04098-6863-4391-90f3-67c8279b4afe-ovnkube-script-lib\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.493128 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:01.491202 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 25 01:11:01.493128 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:01.491265 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret podName:62f24f02-b14f-4633-a4e0-bed308f33ada nodeName:}" failed. No retries permitted until 2026-07-25 01:11:01.991247232 +0000 UTC m=+3.112733075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret") pod "global-pull-secret-syncer-jm7hh" (UID: "62f24f02-b14f-4633-a4e0-bed308f33ada") : object "kube-system"/"original-pull-secret" not registered Jul 25 01:11:01.493128 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.491296 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.493128 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.491510 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6-konnectivity-ca\") pod \"konnectivity-agent-fkgvk\" (UID: \"ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6\") " pod="kube-system/konnectivity-agent-fkgvk" Jul 25 01:11:01.493250 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.493175 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6-agent-certs\") pod \"konnectivity-agent-fkgvk\" (UID: \"ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6\") " pod="kube-system/konnectivity-agent-fkgvk" Jul 25 01:11:01.493502 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.493485 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aed04098-6863-4391-90f3-67c8279b4afe-ovn-node-metrics-cert\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.498171 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.498148 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk2kd\" (UniqueName: \"kubernetes.io/projected/becef38c-52d9-4baa-bfe9-30f712afabdc-kube-api-access-xk2kd\") pod \"node-resolver-9sn77\" (UID: \"becef38c-52d9-4baa-bfe9-30f712afabdc\") " pod="openshift-dns/node-resolver-9sn77" Jul 25 01:11:01.499818 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.499784 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxmbr\" (UniqueName: \"kubernetes.io/projected/684c7548-f719-4ab6-a653-3e95e6511d2e-kube-api-access-sxmbr\") pod \"multus-cgj8z\" (UID: \"684c7548-f719-4ab6-a653-3e95e6511d2e\") " pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.499938 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.499824 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ndjt\" (UniqueName: \"kubernetes.io/projected/187c72b1-1094-47c9-bb23-cbb30ab6ca4c-kube-api-access-4ndjt\") pod \"multus-additional-cni-plugins-xhsbs\" (UID: \"187c72b1-1094-47c9-bb23-cbb30ab6ca4c\") " pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.500201 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.500161 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrqxz\" (UniqueName: \"kubernetes.io/projected/145204ec-0354-45f7-846e-95e3e5580c70-kube-api-access-lrqxz\") pod \"iptables-alerter-5jxlb\" (UID: \"145204ec-0354-45f7-846e-95e3e5580c70\") " pod="openshift-network-operator/iptables-alerter-5jxlb" Jul 25 01:11:01.501210 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.501194 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttq4m\" (UniqueName: \"kubernetes.io/projected/aed04098-6863-4391-90f3-67c8279b4afe-kube-api-access-ttq4m\") pod \"ovnkube-node-lph8b\" (UID: \"aed04098-6863-4391-90f3-67c8279b4afe\") " pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.584174 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.584147 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" Jul 25 01:11:01.593937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.593911 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" Jul 25 01:11:01.603328 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.603309 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tb5g6" Jul 25 01:11:01.607310 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.607294 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xhsbs" Jul 25 01:11:01.613815 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.613780 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:01.622397 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.622378 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fkgvk" Jul 25 01:11:01.629034 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.629015 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9sn77" Jul 25 01:11:01.638525 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.638508 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5jxlb" Jul 25 01:11:01.643143 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.643127 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cgj8z" Jul 25 01:11:01.764103 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.764041 2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 25 01:11:01.894648 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.894623 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs\") pod \"network-metrics-daemon-xtmst\" (UID: \"55676d4a-8fb7-483b-b880-232cc2d10465\") " pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:01.894840 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:01.894820 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 25 01:11:01.894977 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:01.894879 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs podName:55676d4a-8fb7-483b-b880-232cc2d10465 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:02.894864387 +0000 UTC m=+4.016350214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs") pod "network-metrics-daemon-xtmst" (UID: "55676d4a-8fb7-483b-b880-232cc2d10465") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 25 01:11:01.995896 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.995868 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6464s\" (UniqueName: \"kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s\") pod \"network-check-target-2498t\" (UID: \"6727d4bf-1e34-43aa-8766-cc4e224468b3\") " pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:01.996039 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:01.995914 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret\") pod \"global-pull-secret-syncer-jm7hh\" (UID: \"62f24f02-b14f-4633-a4e0-bed308f33ada\") " pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:01.996039 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:01.996029 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 25 01:11:01.996146 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:01.996044 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 25 01:11:01.996146 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:01.996062 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 25 01:11:01.996146 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:01.996074 2554 projected.go:196] Error preparing data for projected volume kube-api-access-6464s for pod openshift-network-diagnostics/network-check-target-2498t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 25 01:11:01.996146 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:01.996099 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret podName:62f24f02-b14f-4633-a4e0-bed308f33ada nodeName:}" failed. No retries permitted until 2026-07-25 01:11:02.99607374 +0000 UTC m=+4.117559588 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret") pod "global-pull-secret-syncer-jm7hh" (UID: "62f24f02-b14f-4633-a4e0-bed308f33ada") : object "kube-system"/"original-pull-secret" not registered Jul 25 01:11:01.996146 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:01.996127 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s podName:6727d4bf-1e34-43aa-8766-cc4e224468b3 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:02.996107659 +0000 UTC m=+4.117593487 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6464s" (UniqueName: "kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s") pod "network-check-target-2498t" (UID: "6727d4bf-1e34-43aa-8766-cc4e224468b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 25 01:11:02.269443 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:02.269419 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187c72b1_1094_47c9_bb23_cbb30ab6ca4c.slice/crio-774d072f127fe45dd6205340713e06e0a3f2d9777ad0144fa8a3b1a634d6c6fb WatchSource:0}: Error finding container 774d072f127fe45dd6205340713e06e0a3f2d9777ad0144fa8a3b1a634d6c6fb: Status 404 returned error can't find the container with id 774d072f127fe45dd6205340713e06e0a3f2d9777ad0144fa8a3b1a634d6c6fb Jul 25 01:11:02.288877 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:02.288845 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684c7548_f719_4ab6_a653_3e95e6511d2e.slice/crio-8cfaaf191b4622389190aa9ac633d87303bc8d0823757bad898c55555a32ec6d WatchSource:0}: Error finding container 8cfaaf191b4622389190aa9ac633d87303bc8d0823757bad898c55555a32ec6d: Status 404 returned error can't find the container with id 8cfaaf191b4622389190aa9ac633d87303bc8d0823757bad898c55555a32ec6d Jul 25 01:11:02.290116 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:02.290094 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1d3e0c_04ea_4e24_ad06_91b0d2bbe0b6.slice/crio-77be8b0fef2072b64b1674c29ebbab96c4d60912b23ea0e094e0732da8a97dfa WatchSource:0}: Error finding container 77be8b0fef2072b64b1674c29ebbab96c4d60912b23ea0e094e0732da8a97dfa: Status 404 returned error can't find the container with id 77be8b0fef2072b64b1674c29ebbab96c4d60912b23ea0e094e0732da8a97dfa Jul 25 01:11:02.290856 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:02.290819 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed04098_6863_4391_90f3_67c8279b4afe.slice/crio-54fb5f7f24ca41b4a55de7dc090f21725e3daec4ab03953e15ff7e37f85e4bbc WatchSource:0}: Error finding container 54fb5f7f24ca41b4a55de7dc090f21725e3daec4ab03953e15ff7e37f85e4bbc: Status 404 returned error can't find the container with id 54fb5f7f24ca41b4a55de7dc090f21725e3daec4ab03953e15ff7e37f85e4bbc Jul 25 01:11:02.291645 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:02.291621 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda1253f_1d94_4dc3_a2ce_5694538260dc.slice/crio-98e8ff651d718bb343203bde9fc28031438f015d5704ebbefd21da16be255d6c WatchSource:0}: Error finding container 98e8ff651d718bb343203bde9fc28031438f015d5704ebbefd21da16be255d6c: Status 404 returned error can't find the container with id 98e8ff651d718bb343203bde9fc28031438f015d5704ebbefd21da16be255d6c Jul 25 01:11:02.297394 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:02.297372 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb6ecc9_98e0_411a_8a63_a5cce9773efb.slice/crio-a7aeef5d677c7b2fdeb5a5cd1764e0e7e6eaf5a1a73b041e307c36f5d2d8ff21 WatchSource:0}: Error finding container a7aeef5d677c7b2fdeb5a5cd1764e0e7e6eaf5a1a73b041e307c36f5d2d8ff21: Status 404 returned error can't find the container with id a7aeef5d677c7b2fdeb5a5cd1764e0e7e6eaf5a1a73b041e307c36f5d2d8ff21 Jul 25 01:11:02.297977 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:02.297837 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd8d67b_61d5_4a13_b319_3a64776c0a5b.slice/crio-9be5b35a4bc14f82da4ca1a0bcef0da46729041d2b1921b46b16f0242bac047e WatchSource:0}: Error finding container 9be5b35a4bc14f82da4ca1a0bcef0da46729041d2b1921b46b16f0242bac047e: Status 404 returned error can't find the container with id 9be5b35a4bc14f82da4ca1a0bcef0da46729041d2b1921b46b16f0242bac047e Jul 25 01:11:02.298673 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:02.298593 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145204ec_0354_45f7_846e_95e3e5580c70.slice/crio-96a704579ee1452f30f2557ea92dabe68f6fcaed82b841e0749864535dbee675 WatchSource:0}: Error finding container 96a704579ee1452f30f2557ea92dabe68f6fcaed82b841e0749864535dbee675: Status 404 returned error can't find the container with id 96a704579ee1452f30f2557ea92dabe68f6fcaed82b841e0749864535dbee675 Jul 25 01:11:02.303925 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:02.301124 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbecef38c_52d9_4baa_bfe9_30f712afabdc.slice/crio-cc32bc9b8c6ffadca996e780aa63faedc1905d9845aefefb674910a4eb7fb2f7 WatchSource:0}: Error finding container cc32bc9b8c6ffadca996e780aa63faedc1905d9845aefefb674910a4eb7fb2f7: Status 404 returned error can't find the container with id cc32bc9b8c6ffadca996e780aa63faedc1905d9845aefefb674910a4eb7fb2f7 Jul 25 01:11:02.313110 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:02.313073 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-24 01:06:01 +0000 UTC" deadline="2028-04-03 22:41:04.114447708 +0000 UTC" Jul 25 01:11:02.313110 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:02.313109 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14853h30m1.801341512s" Jul 25 01:11:02.375249 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:02.375099 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:02.375391 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:02.375359 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm7hh" podUID="62f24f02-b14f-4633-a4e0-bed308f33ada" Jul 25 01:11:02.387775 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:02.387742 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal" event={"ID":"a1bc6362a334464c48c83d833f37a3ca","Type":"ContainerStarted","Data":"6da4131bdec82ca8b7809838e95714ab7b1adcf78d8207377159c6e03159f862"} Jul 25 01:11:02.389813 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:02.389768 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tb5g6" event={"ID":"dfd8d67b-61d5-4a13-b319-3a64776c0a5b","Type":"ContainerStarted","Data":"9be5b35a4bc14f82da4ca1a0bcef0da46729041d2b1921b46b16f0242bac047e"} Jul 25 01:11:02.393094 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:02.391691 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9sn77" event={"ID":"becef38c-52d9-4baa-bfe9-30f712afabdc","Type":"ContainerStarted","Data":"cc32bc9b8c6ffadca996e780aa63faedc1905d9845aefefb674910a4eb7fb2f7"} Jul 25 01:11:02.393923 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:02.393901 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5jxlb" event={"ID":"145204ec-0354-45f7-846e-95e3e5580c70","Type":"ContainerStarted","Data":"96a704579ee1452f30f2557ea92dabe68f6fcaed82b841e0749864535dbee675"} Jul 25 01:11:02.395249 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:02.395225 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhsbs" event={"ID":"187c72b1-1094-47c9-bb23-cbb30ab6ca4c","Type":"ContainerStarted","Data":"774d072f127fe45dd6205340713e06e0a3f2d9777ad0144fa8a3b1a634d6c6fb"} Jul 25 01:11:02.396472 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:02.396439 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" event={"ID":"1bb6ecc9-98e0-411a-8a63-a5cce9773efb","Type":"ContainerStarted","Data":"a7aeef5d677c7b2fdeb5a5cd1764e0e7e6eaf5a1a73b041e307c36f5d2d8ff21"} Jul 25 01:11:02.397653 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:02.397627 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" event={"ID":"fda1253f-1d94-4dc3-a2ce-5694538260dc","Type":"ContainerStarted","Data":"98e8ff651d718bb343203bde9fc28031438f015d5704ebbefd21da16be255d6c"} Jul 25 01:11:02.398794 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:02.398774 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" event={"ID":"aed04098-6863-4391-90f3-67c8279b4afe","Type":"ContainerStarted","Data":"54fb5f7f24ca41b4a55de7dc090f21725e3daec4ab03953e15ff7e37f85e4bbc"} Jul 25 01:11:02.399816 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:02.399780 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cgj8z" event={"ID":"684c7548-f719-4ab6-a653-3e95e6511d2e","Type":"ContainerStarted","Data":"8cfaaf191b4622389190aa9ac633d87303bc8d0823757bad898c55555a32ec6d"} Jul 25 01:11:02.400897 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:02.400849 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal" podStartSLOduration=2.40083585 podStartE2EDuration="2.40083585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:11:02.40010028 +0000 UTC m=+3.521586127" watchObservedRunningTime="2026-07-25 01:11:02.40083585 +0000 UTC m=+3.522321700" Jul 25 01:11:02.401019 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:02.400998 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fkgvk" event={"ID":"ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6","Type":"ContainerStarted","Data":"77be8b0fef2072b64b1674c29ebbab96c4d60912b23ea0e094e0732da8a97dfa"} Jul 25 01:11:02.902389 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:02.901572 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs\") pod \"network-metrics-daemon-xtmst\" (UID: \"55676d4a-8fb7-483b-b880-232cc2d10465\") " pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:02.902389 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:02.901873 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 25 01:11:02.902389 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:02.901939 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs podName:55676d4a-8fb7-483b-b880-232cc2d10465 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:04.901919973 +0000 UTC m=+6.023405826 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs") pod "network-metrics-daemon-xtmst" (UID: "55676d4a-8fb7-483b-b880-232cc2d10465") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 25 01:11:03.002738 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:03.002699 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6464s\" (UniqueName: \"kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s\") pod \"network-check-target-2498t\" (UID: \"6727d4bf-1e34-43aa-8766-cc4e224468b3\") " pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:03.002940 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:03.002766 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret\") pod \"global-pull-secret-syncer-jm7hh\" (UID: \"62f24f02-b14f-4633-a4e0-bed308f33ada\") " pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:03.003000 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:03.002960 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 25 01:11:03.003050 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:03.003028 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret podName:62f24f02-b14f-4633-a4e0-bed308f33ada nodeName:}" failed. No retries permitted until 2026-07-25 01:11:05.003010361 +0000 UTC m=+6.124496202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret") pod "global-pull-secret-syncer-jm7hh" (UID: "62f24f02-b14f-4633-a4e0-bed308f33ada") : object "kube-system"/"original-pull-secret" not registered Jul 25 01:11:03.003566 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:03.003545 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 25 01:11:03.003636 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:03.003573 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 25 01:11:03.003636 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:03.003587 2554 projected.go:196] Error preparing data for projected volume kube-api-access-6464s for pod openshift-network-diagnostics/network-check-target-2498t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 25 01:11:03.003636 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:03.003634 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s podName:6727d4bf-1e34-43aa-8766-cc4e224468b3 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:05.003618757 +0000 UTC m=+6.125104600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6464s" (UniqueName: "kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s") pod "network-check-target-2498t" (UID: "6727d4bf-1e34-43aa-8766-cc4e224468b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 25 01:11:03.313815 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:03.313694 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-24 01:06:01 +0000 UTC" deadline="2028-02-03 18:29:21.372726455 +0000 UTC" Jul 25 01:11:03.313815 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:03.313730 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13409h18m18.059000829s" Jul 25 01:11:03.381306 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:03.376991 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:03.381306 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:03.377112 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2498t" podUID="6727d4bf-1e34-43aa-8766-cc4e224468b3" Jul 25 01:11:03.381306 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:03.377471 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:03.381306 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:03.377568 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtmst" podUID="55676d4a-8fb7-483b-b880-232cc2d10465" Jul 25 01:11:03.413340 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:03.412781 2554 generic.go:356] "Generic (PLEG): container finished" podID="b8fdbf3fd61f65533c9d5667ec910c4a" containerID="cf7e17bc6407e06b26297dab8786756c9d39c39e1a5f86bca0b2233a9d49bc93" exitCode=0 Jul 25 01:11:03.413340 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:03.412912 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal" event={"ID":"b8fdbf3fd61f65533c9d5667ec910c4a","Type":"ContainerDied","Data":"cf7e17bc6407e06b26297dab8786756c9d39c39e1a5f86bca0b2233a9d49bc93"} Jul 25 01:11:04.375424 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:04.375394 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:04.375963 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:04.375548 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm7hh" podUID="62f24f02-b14f-4633-a4e0-bed308f33ada" Jul 25 01:11:04.428543 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:04.428508 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal" event={"ID":"b8fdbf3fd61f65533c9d5667ec910c4a","Type":"ContainerStarted","Data":"54af41ff7f0352b9a804be6c46e4d523c0ef3ac125550bcf0971a47b9d64cc8f"} Jul 25 01:11:04.453568 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:04.452678 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal" podStartSLOduration=4.452657846 podStartE2EDuration="4.452657846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:11:04.445034124 +0000 UTC m=+5.566519976" watchObservedRunningTime="2026-07-25 01:11:04.452657846 +0000 UTC m=+5.574143698" Jul 25 01:11:04.921222 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:04.921151 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs\") pod \"network-metrics-daemon-xtmst\" (UID: \"55676d4a-8fb7-483b-b880-232cc2d10465\") " pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:04.921454 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:04.921299 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 25 01:11:04.921454 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:04.921356 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs podName:55676d4a-8fb7-483b-b880-232cc2d10465 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:08.921341356 +0000 UTC m=+10.042827184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs") pod "network-metrics-daemon-xtmst" (UID: "55676d4a-8fb7-483b-b880-232cc2d10465") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 25 01:11:05.021926 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:05.021889 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6464s\" (UniqueName: \"kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s\") pod \"network-check-target-2498t\" (UID: \"6727d4bf-1e34-43aa-8766-cc4e224468b3\") " pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:05.022103 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:05.021951 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret\") pod \"global-pull-secret-syncer-jm7hh\" (UID: \"62f24f02-b14f-4633-a4e0-bed308f33ada\") " pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:05.022169 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:05.022107 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 25 01:11:05.022169 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:05.022166 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret podName:62f24f02-b14f-4633-a4e0-bed308f33ada nodeName:}" failed. No retries permitted until 2026-07-25 01:11:09.022148782 +0000 UTC m=+10.143634615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret") pod "global-pull-secret-syncer-jm7hh" (UID: "62f24f02-b14f-4633-a4e0-bed308f33ada") : object "kube-system"/"original-pull-secret" not registered Jul 25 01:11:05.022275 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:05.022253 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 25 01:11:05.022275 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:05.022266 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 25 01:11:05.022370 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:05.022280 2554 projected.go:196] Error preparing data for projected volume kube-api-access-6464s for pod openshift-network-diagnostics/network-check-target-2498t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 25 01:11:05.022370 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:05.022315 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s podName:6727d4bf-1e34-43aa-8766-cc4e224468b3 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:09.022303318 +0000 UTC m=+10.143789151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6464s" (UniqueName: "kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s") pod "network-check-target-2498t" (UID: "6727d4bf-1e34-43aa-8766-cc4e224468b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 25 01:11:05.381317 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:05.381249 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:05.381746 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:05.381376 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtmst" podUID="55676d4a-8fb7-483b-b880-232cc2d10465" Jul 25 01:11:05.381746 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:05.381729 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:05.381892 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:05.381829 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2498t" podUID="6727d4bf-1e34-43aa-8766-cc4e224468b3" Jul 25 01:11:06.374929 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:06.374894 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:06.375098 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:06.375017 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm7hh" podUID="62f24f02-b14f-4633-a4e0-bed308f33ada" Jul 25 01:11:07.375094 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:07.375014 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:07.375524 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:07.375193 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2498t" podUID="6727d4bf-1e34-43aa-8766-cc4e224468b3" Jul 25 01:11:07.375524 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:07.375209 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:07.375524 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:07.375350 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtmst" podUID="55676d4a-8fb7-483b-b880-232cc2d10465" Jul 25 01:11:08.375066 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:08.375029 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:08.375254 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:08.375150 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm7hh" podUID="62f24f02-b14f-4633-a4e0-bed308f33ada" Jul 25 01:11:08.954362 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:08.954324 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs\") pod \"network-metrics-daemon-xtmst\" (UID: \"55676d4a-8fb7-483b-b880-232cc2d10465\") " pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:08.954547 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:08.954459 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 25 01:11:08.954547 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:08.954519 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs podName:55676d4a-8fb7-483b-b880-232cc2d10465 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:16.954500968 +0000 UTC m=+18.075986797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs") pod "network-metrics-daemon-xtmst" (UID: "55676d4a-8fb7-483b-b880-232cc2d10465") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 25 01:11:09.056097 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:09.055339 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6464s\" (UniqueName: \"kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s\") pod \"network-check-target-2498t\" (UID: \"6727d4bf-1e34-43aa-8766-cc4e224468b3\") " pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:09.056097 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:09.055394 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret\") pod \"global-pull-secret-syncer-jm7hh\" (UID: \"62f24f02-b14f-4633-a4e0-bed308f33ada\") " pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:09.056097 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:09.055528 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 25 01:11:09.056097 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:09.055584 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret podName:62f24f02-b14f-4633-a4e0-bed308f33ada nodeName:}" failed. No retries permitted until 2026-07-25 01:11:17.0555666 +0000 UTC m=+18.177052432 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret") pod "global-pull-secret-syncer-jm7hh" (UID: "62f24f02-b14f-4633-a4e0-bed308f33ada") : object "kube-system"/"original-pull-secret" not registered Jul 25 01:11:09.056097 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:09.056003 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 25 01:11:09.056097 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:09.056023 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 25 01:11:09.056097 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:09.056038 2554 projected.go:196] Error preparing data for projected volume kube-api-access-6464s for pod openshift-network-diagnostics/network-check-target-2498t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 25 01:11:09.056097 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:09.056071 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s podName:6727d4bf-1e34-43aa-8766-cc4e224468b3 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:17.056061537 +0000 UTC m=+18.177547378 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6464s" (UniqueName: "kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s") pod "network-check-target-2498t" (UID: "6727d4bf-1e34-43aa-8766-cc4e224468b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 25 01:11:09.376571 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:09.376501 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:09.377024 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:09.376608 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtmst" podUID="55676d4a-8fb7-483b-b880-232cc2d10465" Jul 25 01:11:09.377024 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:09.376947 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:09.377024 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:09.377015 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2498t" podUID="6727d4bf-1e34-43aa-8766-cc4e224468b3" Jul 25 01:11:10.375543 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:10.375509 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:10.375689 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:10.375637 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm7hh" podUID="62f24f02-b14f-4633-a4e0-bed308f33ada" Jul 25 01:11:11.376752 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:11.376721 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:11.377221 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:11.376892 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtmst" podUID="55676d4a-8fb7-483b-b880-232cc2d10465" Jul 25 01:11:11.377283 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:11.377255 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:11.377360 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:11.377342 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2498t" podUID="6727d4bf-1e34-43aa-8766-cc4e224468b3" Jul 25 01:11:12.375555 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:12.375525 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:12.375741 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:12.375640 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm7hh" podUID="62f24f02-b14f-4633-a4e0-bed308f33ada" Jul 25 01:11:13.375116 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:13.375038 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:13.375572 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:13.375054 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:13.375572 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:13.375182 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtmst" podUID="55676d4a-8fb7-483b-b880-232cc2d10465" Jul 25 01:11:13.375572 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:13.375215 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2498t" podUID="6727d4bf-1e34-43aa-8766-cc4e224468b3" Jul 25 01:11:14.375673 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:14.375641 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:14.376159 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:14.375756 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm7hh" podUID="62f24f02-b14f-4633-a4e0-bed308f33ada" Jul 25 01:11:15.375236 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:15.375204 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:15.375399 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:15.375327 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2498t" podUID="6727d4bf-1e34-43aa-8766-cc4e224468b3" Jul 25 01:11:15.375399 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:15.375378 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:15.375528 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:15.375508 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtmst" podUID="55676d4a-8fb7-483b-b880-232cc2d10465" Jul 25 01:11:16.375751 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:16.375719 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:16.376243 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:16.375856 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm7hh" podUID="62f24f02-b14f-4633-a4e0-bed308f33ada" Jul 25 01:11:17.015576 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:17.015548 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs\") pod \"network-metrics-daemon-xtmst\" (UID: \"55676d4a-8fb7-483b-b880-232cc2d10465\") " pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:17.015708 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:17.015641 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 25 01:11:17.015708 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:17.015699 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs podName:55676d4a-8fb7-483b-b880-232cc2d10465 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:33.015681845 +0000 UTC m=+34.137167673 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs") pod "network-metrics-daemon-xtmst" (UID: "55676d4a-8fb7-483b-b880-232cc2d10465") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 25 01:11:17.116476 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:17.116437 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret\") pod \"global-pull-secret-syncer-jm7hh\" (UID: \"62f24f02-b14f-4633-a4e0-bed308f33ada\") " pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:17.116630 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:17.116507 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6464s\" (UniqueName: \"kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s\") pod \"network-check-target-2498t\" (UID: \"6727d4bf-1e34-43aa-8766-cc4e224468b3\") " pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:17.116630 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:17.116581 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 25 01:11:17.116630 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:17.116606 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 25 01:11:17.116630 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:17.116617 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 25 01:11:17.116630 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:17.116626 2554 projected.go:196] Error preparing data for projected volume kube-api-access-6464s for pod openshift-network-diagnostics/network-check-target-2498t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 25 01:11:17.116844 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:17.116645 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret podName:62f24f02-b14f-4633-a4e0-bed308f33ada nodeName:}" failed. No retries permitted until 2026-07-25 01:11:33.116629323 +0000 UTC m=+34.238115151 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret") pod "global-pull-secret-syncer-jm7hh" (UID: "62f24f02-b14f-4633-a4e0-bed308f33ada") : object "kube-system"/"original-pull-secret" not registered Jul 25 01:11:17.116844 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:17.116665 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s podName:6727d4bf-1e34-43aa-8766-cc4e224468b3 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:33.116653364 +0000 UTC m=+34.238139191 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6464s" (UniqueName: "kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s") pod "network-check-target-2498t" (UID: "6727d4bf-1e34-43aa-8766-cc4e224468b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 25 01:11:17.379314 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:17.379223 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:17.379721 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:17.379357 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtmst" podUID="55676d4a-8fb7-483b-b880-232cc2d10465" Jul 25 01:11:17.379721 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:17.379420 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:17.379721 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:17.379516 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2498t" podUID="6727d4bf-1e34-43aa-8766-cc4e224468b3" Jul 25 01:11:18.375322 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:18.375287 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:18.375509 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:18.375406 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm7hh" podUID="62f24f02-b14f-4633-a4e0-bed308f33ada" Jul 25 01:11:19.378896 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:19.378858 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:19.379311 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:19.379041 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2498t" podUID="6727d4bf-1e34-43aa-8766-cc4e224468b3" Jul 25 01:11:19.379311 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:19.379152 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:19.379430 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:19.379351 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtmst" podUID="55676d4a-8fb7-483b-b880-232cc2d10465" Jul 25 01:11:20.375115 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:20.375079 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:20.375264 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:20.375189 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm7hh" podUID="62f24f02-b14f-4633-a4e0-bed308f33ada" Jul 25 01:11:21.375651 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:21.374977 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:21.375651 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:21.375498 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2498t" podUID="6727d4bf-1e34-43aa-8766-cc4e224468b3" Jul 25 01:11:21.375651 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:21.374999 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:21.375651 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:21.375620 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtmst" podUID="55676d4a-8fb7-483b-b880-232cc2d10465" Jul 25 01:11:21.464205 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:21.464178 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fkgvk" event={"ID":"ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6","Type":"ContainerStarted","Data":"9b1efe20175525c115f3064f45585dd64451a960ab0f8de8baa8ca2f9bc249ea"} Jul 25 01:11:21.465550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:21.465530 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tb5g6" event={"ID":"dfd8d67b-61d5-4a13-b319-3a64776c0a5b","Type":"ContainerStarted","Data":"1aba01f6376a511cf3ab02de3982ad6aaf8558bd71c9406f3378d2b9fa657531"} Jul 25 01:11:21.466896 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:21.466877 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9sn77" event={"ID":"becef38c-52d9-4baa-bfe9-30f712afabdc","Type":"ContainerStarted","Data":"d623b0d1dca23dbd2f67f0490218401167653a2d5d4dd7e40097ae1a5073c8e1"} Jul 25 01:11:21.468273 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:21.468253 2554 generic.go:356] "Generic (PLEG): container finished" podID="187c72b1-1094-47c9-bb23-cbb30ab6ca4c" containerID="b282dca9730ac4932cf9209445ff36f52b839c79cb055e2e76c4caf7d0de82ed" exitCode=0 Jul 25 01:11:21.468344 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:21.468282 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhsbs" event={"ID":"187c72b1-1094-47c9-bb23-cbb30ab6ca4c","Type":"ContainerDied","Data":"b282dca9730ac4932cf9209445ff36f52b839c79cb055e2e76c4caf7d0de82ed"} Jul 25 01:11:21.469830 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:21.469790 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" event={"ID":"1bb6ecc9-98e0-411a-8a63-a5cce9773efb","Type":"ContainerStarted","Data":"07fc2708e070218012d853b19cca142e8f2314627d6e223821da1cbb1f05ced4"} Jul 25 01:11:21.471078 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:21.471061 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" event={"ID":"fda1253f-1d94-4dc3-a2ce-5694538260dc","Type":"ContainerStarted","Data":"af1b605686748c2995f88fde609e3806e1d96882467011b06cfab8f206b1a0ba"} Jul 25 01:11:21.473295 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:21.473275 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" event={"ID":"aed04098-6863-4391-90f3-67c8279b4afe","Type":"ContainerStarted","Data":"11a0544680e406c77c902c6e5c318ddd4ee03694368fd3882351269466cdd9f9"} Jul 25 01:11:21.473374 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:21.473334 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" event={"ID":"aed04098-6863-4391-90f3-67c8279b4afe","Type":"ContainerStarted","Data":"1a5ddef4e134c4e4649f2f5e4107c9eea0ec67902cf50c07dc47a39df857e58d"} Jul 25 01:11:21.473417 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:21.473373 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" event={"ID":"aed04098-6863-4391-90f3-67c8279b4afe","Type":"ContainerStarted","Data":"7211185e5f6804250913703da4492cd775361296049065eb8884b84a58a41175"} Jul 25 01:11:21.473417 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:21.473408 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" event={"ID":"aed04098-6863-4391-90f3-67c8279b4afe","Type":"ContainerStarted","Data":"c440df72a1e04d18a15557eb3ec7befef00d0bf329bf759ccbd462247de1ddc4"} Jul 25 01:11:21.474601 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:21.474584 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cgj8z" event={"ID":"684c7548-f719-4ab6-a653-3e95e6511d2e","Type":"ContainerStarted","Data":"a373c5d3ffb5b90eb755c433a109666d4c971fa5a9f4b9eaaa827cccd717cabb"} Jul 25 01:11:21.481179 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:21.481144 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fkgvk" podStartSLOduration=11.794180548 podStartE2EDuration="22.481132366s" totalImagesPullingTime="10.686951818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:11:21.480977271 +0000 UTC m=+22.602463121" watchObservedRunningTime="2026-07-25 01:11:21.481132366 +0000 UTC m=+22.602618216" Jul 25 01:11:21.494684 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:21.494648 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tb5g6" podStartSLOduration=3.782895104 podStartE2EDuration="22.494636741s" totalImagesPullingTime="18.711741637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:11:21.494260278 +0000 UTC m=+22.615746128" watchObservedRunningTime="2026-07-25 01:11:21.494636741 +0000 UTC m=+22.616122590" Jul 25 01:11:21.558817 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:21.558764 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9sn77" podStartSLOduration=3.850366531 podStartE2EDuration="22.558748005s" totalImagesPullingTime="18.708381474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:11:21.538467572 +0000 UTC m=+22.659953421" watchObservedRunningTime="2026-07-25 01:11:21.558748005 +0000 UTC m=+22.680233854" Jul 25 01:11:21.559236 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:21.559209 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8lkx2" podStartSLOduration=3.825703449 podStartE2EDuration="22.559198143s" totalImagesPullingTime="18.733494694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:11:21.55896189 +0000 UTC m=+22.680447740" watchObservedRunningTime="2026-07-25 01:11:21.559198143 +0000 UTC m=+22.680683995" Jul 25 01:11:21.576617 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:21.576582 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cgj8z" podStartSLOduration=3.7956938 podStartE2EDuration="22.57657098s" totalImagesPullingTime="18.78087718s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:11:21.576483161 +0000 UTC m=+22.697969012" watchObservedRunningTime="2026-07-25 01:11:21.57657098 +0000 UTC m=+22.698056829" Jul 25 01:11:22.368480 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:22.368433 2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 25 01:11:22.374835 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:22.374789 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:22.374943 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:22.374912 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm7hh" podUID="62f24f02-b14f-4633-a4e0-bed308f33ada" Jul 25 01:11:22.479011 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:22.478983 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" event={"ID":"fda1253f-1d94-4dc3-a2ce-5694538260dc","Type":"ContainerStarted","Data":"738c1b4c23c79315e6f394ca44305a3aabebf83732fd5742f98db970eb0560ea"} Jul 25 01:11:22.482079 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:22.482051 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" event={"ID":"aed04098-6863-4391-90f3-67c8279b4afe","Type":"ContainerStarted","Data":"e57f1ed11d7e91ca82b053820dec1dc504f672b2c2e65066ceb20692e3f516b3"} Jul 25 01:11:22.482188 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:22.482131 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" event={"ID":"aed04098-6863-4391-90f3-67c8279b4afe","Type":"ContainerStarted","Data":"452f8cceb7d18d20cbe07d330f33f458242dc1c5b6af0a5b4e76c41ccc8b072f"} Jul 25 01:11:23.349816 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:23.349669 2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-25T01:11:22.368465742Z","UUID":"663b62a3-a0f3-4b0a-b033-c0f5e1b723ee","Handler":null,"Name":"","Endpoint":""} Jul 25 01:11:23.351644 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:23.351610 2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 25 01:11:23.351790 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:23.351661 2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 25 01:11:23.376135 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:23.375641 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:23.376135 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:23.375647 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:23.376135 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:23.375830 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtmst" podUID="55676d4a-8fb7-483b-b880-232cc2d10465" Jul 25 01:11:23.376135 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:23.375874 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2498t" podUID="6727d4bf-1e34-43aa-8766-cc4e224468b3" Jul 25 01:11:23.485034 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:23.485002 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5jxlb" event={"ID":"145204ec-0354-45f7-846e-95e3e5580c70","Type":"ContainerStarted","Data":"d80744075ac5a3a9785c9982e9c335f89c8e187cc0d6ed86f3d4ef3112abb207"} Jul 25 01:11:23.500133 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:23.499921 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5jxlb" podStartSLOduration=5.790839329 podStartE2EDuration="24.499906758s" totalImagesPullingTime="18.709067429s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:11:23.499655105 +0000 UTC m=+24.621140956" watchObservedRunningTime="2026-07-25 01:11:23.499906758 +0000 UTC m=+24.621392608" Jul 25 01:11:24.375817 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:24.375769 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:24.375995 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:24.375923 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm7hh" podUID="62f24f02-b14f-4633-a4e0-bed308f33ada" Jul 25 01:11:24.489318 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:24.489282 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" event={"ID":"fda1253f-1d94-4dc3-a2ce-5694538260dc","Type":"ContainerStarted","Data":"7adc1d1af8be8c90070e24e002e0127f2ab0822fe6c18430047245113cfaf9b4"} Jul 25 01:11:24.493213 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:24.493182 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" event={"ID":"aed04098-6863-4391-90f3-67c8279b4afe","Type":"ContainerStarted","Data":"7adb085589b48dab130c9cd65ce4789416a1d1fde4ec21285631aaeecf3086a5"} Jul 25 01:11:24.510208 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:24.510155 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lffgj" podStartSLOduration=4.617635043 podStartE2EDuration="25.510137324s" totalImagesPullingTime="20.892502281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:10:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:11:24.509755661 +0000 UTC m=+25.631241557" watchObservedRunningTime="2026-07-25 01:11:24.510137324 +0000 UTC m=+25.631623175" Jul 25 01:11:25.090677 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:25.090517 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:25.180055 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:25.180020 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:25.375576 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:25.375497 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:25.375576 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:25.375521 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:25.375785 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:25.375640 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtmst" podUID="55676d4a-8fb7-483b-b880-232cc2d10465" Jul 25 01:11:25.375851 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:25.375786 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2498t" podUID="6727d4bf-1e34-43aa-8766-cc4e224468b3" Jul 25 01:11:25.445367 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:25.445339 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:25.545056 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:25.545020 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:26.063726 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:26.063688 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fkgvk" Jul 25 01:11:26.064359 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:26.064337 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fkgvk" Jul 25 01:11:26.375486 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:26.375407 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:26.375665 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:26.375533 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm7hh" podUID="62f24f02-b14f-4633-a4e0-bed308f33ada" Jul 25 01:11:26.498369 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:26.498327 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fkgvk" Jul 25 01:11:26.498877 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:26.498859 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fkgvk" Jul 25 01:11:26.559836 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:26.559765 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:27.242677 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:27.242471 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:27.375185 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:27.375157 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:27.375343 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:27.375257 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2498t" podUID="6727d4bf-1e34-43aa-8766-cc4e224468b3" Jul 25 01:11:27.375343 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:27.375326 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:27.375474 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:27.375452 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtmst" podUID="55676d4a-8fb7-483b-b880-232cc2d10465" Jul 25 01:11:27.503224 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:27.503137 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" event={"ID":"aed04098-6863-4391-90f3-67c8279b4afe","Type":"ContainerStarted","Data":"649920d3d10845815e198aabfcff4214db6eeff1fc7959cfb95b35ad8ef5bcdb"} Jul 25 01:11:27.503396 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:27.503374 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:27.503509 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:27.503495 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:27.503582 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:27.503545 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:27.505510 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:27.505484 2554 generic.go:356] "Generic (PLEG): container finished" podID="187c72b1-1094-47c9-bb23-cbb30ab6ca4c" containerID="5b679fb58ba9ffd3c1c09422bd0e2e695688f79aadda45648b8ded3a249d508c" exitCode=0 Jul 25 01:11:27.505621 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:27.505553 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhsbs" event={"ID":"187c72b1-1094-47c9-bb23-cbb30ab6ca4c","Type":"ContainerDied","Data":"5b679fb58ba9ffd3c1c09422bd0e2e695688f79aadda45648b8ded3a249d508c"} Jul 25 01:11:27.519906 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:27.519879 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:27.520451 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:27.520435 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:27.530715 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:27.530673 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" podStartSLOduration=9.717901193 podStartE2EDuration="28.530662468s" totalImagesPullingTime="18.812761275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:11:27.529513158 +0000 UTC m=+28.650999001" watchObservedRunningTime="2026-07-25 01:11:27.530662468 +0000 UTC m=+28.652148356" Jul 25 01:11:27.641991 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:27.641966 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:27.742305 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:27.742271 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:27.894072 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:27.894007 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:27.950002 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:27.949973 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:28.375750 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:28.375689 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:28.375896 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:28.375784 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm7hh" podUID="62f24f02-b14f-4633-a4e0-bed308f33ada" Jul 25 01:11:28.452822 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:28.452766 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:28.552050 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:28.552021 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:28.842574 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:28.842544 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:28.894331 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:28.894278 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:29.005673 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:29.005643 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jm7hh"] Jul 25 01:11:29.005843 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:29.005737 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:29.005909 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:29.005836 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm7hh" podUID="62f24f02-b14f-4633-a4e0-bed308f33ada" Jul 25 01:11:29.008820 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:29.008771 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2498t"] Jul 25 01:11:29.008946 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:29.008885 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:29.009013 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:29.008980 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2498t" podUID="6727d4bf-1e34-43aa-8766-cc4e224468b3" Jul 25 01:11:29.009442 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:29.009415 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xtmst"] Jul 25 01:11:29.009550 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:29.009523 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:29.009632 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:29.009613 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtmst" podUID="55676d4a-8fb7-483b-b880-232cc2d10465" Jul 25 01:11:29.511599 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:29.511566 2554 generic.go:356] "Generic (PLEG): container finished" podID="187c72b1-1094-47c9-bb23-cbb30ab6ca4c" containerID="641ac56bebc7364542b3806c11fdab37e661fdf7af0816854615bf4624a4c125" exitCode=0 Jul 25 01:11:29.511752 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:29.511651 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhsbs" event={"ID":"187c72b1-1094-47c9-bb23-cbb30ab6ca4c","Type":"ContainerDied","Data":"641ac56bebc7364542b3806c11fdab37e661fdf7af0816854615bf4624a4c125"} Jul 25 01:11:30.045529 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:30.045504 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:30.102825 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:30.102781 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:30.375402 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:30.375337 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:30.375544 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:30.375436 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2498t" podUID="6727d4bf-1e34-43aa-8766-cc4e224468b3" Jul 25 01:11:30.516450 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:30.516291 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhsbs" event={"ID":"187c72b1-1094-47c9-bb23-cbb30ab6ca4c","Type":"ContainerStarted","Data":"5d7031ff90bd781e2034b17edb40e83ecd8bd61d02399b0505eda8eac6f0ec24"} Jul 25 01:11:31.375146 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:31.375119 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:31.375685 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:31.375228 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtmst" podUID="55676d4a-8fb7-483b-b880-232cc2d10465" Jul 25 01:11:31.375685 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:31.375280 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:31.375685 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:31.375368 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm7hh" podUID="62f24f02-b14f-4633-a4e0-bed308f33ada" Jul 25 01:11:31.520729 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:31.520699 2554 generic.go:356] "Generic (PLEG): container finished" podID="187c72b1-1094-47c9-bb23-cbb30ab6ca4c" containerID="5d7031ff90bd781e2034b17edb40e83ecd8bd61d02399b0505eda8eac6f0ec24" exitCode=0 Jul 25 01:11:31.520889 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:31.520735 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhsbs" event={"ID":"187c72b1-1094-47c9-bb23-cbb30ab6ca4c","Type":"ContainerDied","Data":"5d7031ff90bd781e2034b17edb40e83ecd8bd61d02399b0505eda8eac6f0ec24"} Jul 25 01:11:32.375179 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:32.375146 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:32.375570 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:32.375259 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2498t" podUID="6727d4bf-1e34-43aa-8766-cc4e224468b3" Jul 25 01:11:33.041420 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.041387 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs\") pod \"network-metrics-daemon-xtmst\" (UID: \"55676d4a-8fb7-483b-b880-232cc2d10465\") " pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:33.041620 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:33.041555 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 25 01:11:33.041620 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:33.041618 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs podName:55676d4a-8fb7-483b-b880-232cc2d10465 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:05.041602035 +0000 UTC m=+66.163087867 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs") pod "network-metrics-daemon-xtmst" (UID: "55676d4a-8fb7-483b-b880-232cc2d10465") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 25 01:11:33.142633 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.142598 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret\") pod \"global-pull-secret-syncer-jm7hh\" (UID: \"62f24f02-b14f-4633-a4e0-bed308f33ada\") " pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:33.142845 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.142700 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6464s\" (UniqueName: \"kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s\") pod \"network-check-target-2498t\" (UID: \"6727d4bf-1e34-43aa-8766-cc4e224468b3\") " pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:33.142845 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:33.142754 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 25 01:11:33.142845 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:33.142823 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 25 01:11:33.142845 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:33.142840 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 25 01:11:33.142845 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:33.142846 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret podName:62f24f02-b14f-4633-a4e0-bed308f33ada nodeName:}" failed. No retries permitted until 2026-07-25 01:12:05.142823749 +0000 UTC m=+66.264309590 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret") pod "global-pull-secret-syncer-jm7hh" (UID: "62f24f02-b14f-4633-a4e0-bed308f33ada") : object "kube-system"/"original-pull-secret" not registered Jul 25 01:11:33.143067 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:33.142853 2554 projected.go:196] Error preparing data for projected volume kube-api-access-6464s for pod openshift-network-diagnostics/network-check-target-2498t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 25 01:11:33.143067 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:33.142892 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s podName:6727d4bf-1e34-43aa-8766-cc4e224468b3 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:05.142879225 +0000 UTC m=+66.264365053 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6464s" (UniqueName: "kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s") pod "network-check-target-2498t" (UID: "6727d4bf-1e34-43aa-8766-cc4e224468b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 25 01:11:33.379191 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.379114 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:33.379572 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:33.379236 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm7hh" podUID="62f24f02-b14f-4633-a4e0-bed308f33ada" Jul 25 01:11:33.379572 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.379303 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:33.379572 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:33.379437 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtmst" podUID="55676d4a-8fb7-483b-b880-232cc2d10465" Jul 25 01:11:33.729675 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.729604 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeReady" Jul 25 01:11:33.729841 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.729770 2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 25 01:11:33.762466 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.762439 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69dbd56bdd-clccr"] Jul 25 01:11:33.766047 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.766024 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.768066 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.768038 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 25 01:11:33.768201 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.768073 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 25 01:11:33.768201 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.768157 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 25 01:11:33.768403 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.768362 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j7vlc\"" Jul 25 01:11:33.773101 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.773067 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l9pbg"] Jul 25 01:11:33.773929 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.773908 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 25 01:11:33.776824 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.776782 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zc4st"] Jul 25 01:11:33.776934 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.776918 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:33.779577 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.779560 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 25 01:11:33.779577 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.779571 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 25 01:11:33.779726 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.779579 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4svzl\"" Jul 25 01:11:33.780612 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.780597 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69dbd56bdd-clccr"] Jul 25 01:11:33.780685 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.780675 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zc4st" Jul 25 01:11:33.783082 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.782417 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 25 01:11:33.783082 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.782510 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 25 01:11:33.783082 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.782527 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 25 01:11:33.783082 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.782533 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-srmhs\"" Jul 25 01:11:33.790528 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.790471 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l9pbg"] Jul 25 01:11:33.791136 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.791115 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zc4st"] Jul 25 01:11:33.812516 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.812493 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:33.848619 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.848587 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-certificates\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.848766 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.848637 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.848766 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.848723 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-ca-trust-extracted\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.848766 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.848750 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-image-registry-private-configuration\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.848961 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.848828 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-bound-sa-token\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.848961 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.848862 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-installation-pull-secrets\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.848961 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.848887 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9n8v\" (UniqueName: \"kubernetes.io/projected/8f47ed40-22e9-42da-8c7a-236132c635f2-kube-api-access-x9n8v\") pod \"dns-default-l9pbg\" (UID: \"8f47ed40-22e9-42da-8c7a-236132c635f2\") " pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:33.848961 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.848931 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f47ed40-22e9-42da-8c7a-236132c635f2-tmp-dir\") pod \"dns-default-l9pbg\" (UID: \"8f47ed40-22e9-42da-8c7a-236132c635f2\") " pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:33.849151 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.848966 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6wwr\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-kube-api-access-q6wwr\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.849151 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.848995 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f47ed40-22e9-42da-8c7a-236132c635f2-config-volume\") pod \"dns-default-l9pbg\" (UID: \"8f47ed40-22e9-42da-8c7a-236132c635f2\") " pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:33.849151 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.849063 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-trusted-ca\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.849151 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.849094 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f47ed40-22e9-42da-8c7a-236132c635f2-metrics-tls\") pod \"dns-default-l9pbg\" (UID: \"8f47ed40-22e9-42da-8c7a-236132c635f2\") " pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:33.871372 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.871317 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:33.950474 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.950438 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-certificates\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.950474 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.950480 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpzth\" (UniqueName: \"kubernetes.io/projected/a8916f19-f4e7-4bbd-a0cb-49150afe5542-kube-api-access-bpzth\") pod \"ingress-canary-zc4st\" (UID: \"a8916f19-f4e7-4bbd-a0cb-49150afe5542\") " pod="openshift-ingress-canary/ingress-canary-zc4st" Jul 25 01:11:33.950687 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.950513 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.950687 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.950584 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-ca-trust-extracted\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.950687 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.950608 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8916f19-f4e7-4bbd-a0cb-49150afe5542-cert\") pod \"ingress-canary-zc4st\" (UID: \"a8916f19-f4e7-4bbd-a0cb-49150afe5542\") " pod="openshift-ingress-canary/ingress-canary-zc4st" Jul 25 01:11:33.950687 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.950656 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-image-registry-private-configuration\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.950906 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.950700 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-bound-sa-token\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.950906 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:33.950706 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 25 01:11:33.950906 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:33.950726 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69dbd56bdd-clccr: secret "image-registry-tls" not found Jul 25 01:11:33.950906 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.950727 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-installation-pull-secrets\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.950906 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.950751 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9n8v\" (UniqueName: \"kubernetes.io/projected/8f47ed40-22e9-42da-8c7a-236132c635f2-kube-api-access-x9n8v\") pod \"dns-default-l9pbg\" (UID: \"8f47ed40-22e9-42da-8c7a-236132c635f2\") " pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:33.950906 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:33.950794 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls podName:c3a5ae83-b4b8-47c6-8757-d1212f5ad638 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:34.450767891 +0000 UTC m=+35.572253736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls") pod "image-registry-69dbd56bdd-clccr" (UID: "c3a5ae83-b4b8-47c6-8757-d1212f5ad638") : secret "image-registry-tls" not found Jul 25 01:11:33.950906 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.950843 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f47ed40-22e9-42da-8c7a-236132c635f2-tmp-dir\") pod \"dns-default-l9pbg\" (UID: \"8f47ed40-22e9-42da-8c7a-236132c635f2\") " pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:33.950906 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.950878 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6wwr\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-kube-api-access-q6wwr\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.950906 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.950906 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f47ed40-22e9-42da-8c7a-236132c635f2-config-volume\") pod \"dns-default-l9pbg\" (UID: \"8f47ed40-22e9-42da-8c7a-236132c635f2\") " pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:33.951318 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.950943 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-ca-trust-extracted\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.951318 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.950951 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-trusted-ca\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.951318 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.950976 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f47ed40-22e9-42da-8c7a-236132c635f2-metrics-tls\") pod \"dns-default-l9pbg\" (UID: \"8f47ed40-22e9-42da-8c7a-236132c635f2\") " pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:33.951318 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:33.951102 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 25 01:11:33.951318 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:33.951143 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f47ed40-22e9-42da-8c7a-236132c635f2-metrics-tls podName:8f47ed40-22e9-42da-8c7a-236132c635f2 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:34.451129776 +0000 UTC m=+35.572615606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f47ed40-22e9-42da-8c7a-236132c635f2-metrics-tls") pod "dns-default-l9pbg" (UID: "8f47ed40-22e9-42da-8c7a-236132c635f2") : secret "dns-default-metrics-tls" not found Jul 25 01:11:33.951318 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.951197 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-certificates\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.951646 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.951623 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f47ed40-22e9-42da-8c7a-236132c635f2-tmp-dir\") pod \"dns-default-l9pbg\" (UID: \"8f47ed40-22e9-42da-8c7a-236132c635f2\") " pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:33.951724 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.951706 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f47ed40-22e9-42da-8c7a-236132c635f2-config-volume\") pod \"dns-default-l9pbg\" (UID: \"8f47ed40-22e9-42da-8c7a-236132c635f2\") " pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:33.952246 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.952212 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-trusted-ca\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.955423 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.955399 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-image-registry-private-configuration\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.955526 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.955435 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-installation-pull-secrets\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.959406 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.959387 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9n8v\" (UniqueName: \"kubernetes.io/projected/8f47ed40-22e9-42da-8c7a-236132c635f2-kube-api-access-x9n8v\") pod \"dns-default-l9pbg\" (UID: \"8f47ed40-22e9-42da-8c7a-236132c635f2\") " pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:33.959511 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.959474 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-bound-sa-token\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:33.959584 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:33.959569 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6wwr\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-kube-api-access-q6wwr\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:34.051879 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:34.051787 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpzth\" (UniqueName: \"kubernetes.io/projected/a8916f19-f4e7-4bbd-a0cb-49150afe5542-kube-api-access-bpzth\") pod \"ingress-canary-zc4st\" (UID: \"a8916f19-f4e7-4bbd-a0cb-49150afe5542\") " pod="openshift-ingress-canary/ingress-canary-zc4st" Jul 25 01:11:34.052033 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:34.051896 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8916f19-f4e7-4bbd-a0cb-49150afe5542-cert\") pod \"ingress-canary-zc4st\" (UID: \"a8916f19-f4e7-4bbd-a0cb-49150afe5542\") " pod="openshift-ingress-canary/ingress-canary-zc4st" Jul 25 01:11:34.054505 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:34.054481 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8916f19-f4e7-4bbd-a0cb-49150afe5542-cert\") pod \"ingress-canary-zc4st\" (UID: \"a8916f19-f4e7-4bbd-a0cb-49150afe5542\") " pod="openshift-ingress-canary/ingress-canary-zc4st" Jul 25 01:11:34.059989 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:34.059961 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpzth\" (UniqueName: \"kubernetes.io/projected/a8916f19-f4e7-4bbd-a0cb-49150afe5542-kube-api-access-bpzth\") pod \"ingress-canary-zc4st\" (UID: \"a8916f19-f4e7-4bbd-a0cb-49150afe5542\") " pod="openshift-ingress-canary/ingress-canary-zc4st" Jul 25 01:11:34.091964 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:34.091927 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zc4st" Jul 25 01:11:34.285508 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:34.285460 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:34.350162 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:34.350081 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:34.375659 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:34.375632 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:11:34.378888 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:34.378865 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f4ttp\"" Jul 25 01:11:34.379146 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:34.379130 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 25 01:11:34.379418 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:34.379403 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 25 01:11:34.455073 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:34.455049 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f47ed40-22e9-42da-8c7a-236132c635f2-metrics-tls\") pod \"dns-default-l9pbg\" (UID: \"8f47ed40-22e9-42da-8c7a-236132c635f2\") " pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:34.455221 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:34.455113 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:34.455221 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:34.455176 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 25 01:11:34.455306 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:34.455240 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f47ed40-22e9-42da-8c7a-236132c635f2-metrics-tls podName:8f47ed40-22e9-42da-8c7a-236132c635f2 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:35.455224515 +0000 UTC m=+36.576710343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f47ed40-22e9-42da-8c7a-236132c635f2-metrics-tls") pod "dns-default-l9pbg" (UID: "8f47ed40-22e9-42da-8c7a-236132c635f2") : secret "dns-default-metrics-tls" not found Jul 25 01:11:34.455306 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:34.455280 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 25 01:11:34.455306 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:34.455299 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69dbd56bdd-clccr: secret "image-registry-tls" not found Jul 25 01:11:34.455418 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:34.455355 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls podName:c3a5ae83-b4b8-47c6-8757-d1212f5ad638 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:35.455338911 +0000 UTC m=+36.576824746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls") pod "image-registry-69dbd56bdd-clccr" (UID: "c3a5ae83-b4b8-47c6-8757-d1212f5ad638") : secret "image-registry-tls" not found Jul 25 01:11:34.493483 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:34.493455 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:34.596614 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:34.596578 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:35.176114 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:35.176079 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:35.276441 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:35.276410 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:35.375579 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:35.375543 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:11:35.375579 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:35.375571 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:11:35.377999 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:35.377973 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 25 01:11:35.378201 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:35.378186 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 25 01:11:35.378358 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:35.378346 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-djkst\"" Jul 25 01:11:35.463605 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:35.463515 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f47ed40-22e9-42da-8c7a-236132c635f2-metrics-tls\") pod \"dns-default-l9pbg\" (UID: \"8f47ed40-22e9-42da-8c7a-236132c635f2\") " pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:35.463605 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:35.463585 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:35.464164 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:35.463681 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 25 01:11:35.464164 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:35.463700 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 25 01:11:35.464164 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:35.463714 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69dbd56bdd-clccr: secret "image-registry-tls" not found Jul 25 01:11:35.464164 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:35.463751 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f47ed40-22e9-42da-8c7a-236132c635f2-metrics-tls podName:8f47ed40-22e9-42da-8c7a-236132c635f2 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:37.463729601 +0000 UTC m=+38.585215433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f47ed40-22e9-42da-8c7a-236132c635f2-metrics-tls") pod "dns-default-l9pbg" (UID: "8f47ed40-22e9-42da-8c7a-236132c635f2") : secret "dns-default-metrics-tls" not found Jul 25 01:11:35.464164 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:35.463773 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls podName:c3a5ae83-b4b8-47c6-8757-d1212f5ad638 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:37.46375834 +0000 UTC m=+38.585244169 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls") pod "image-registry-69dbd56bdd-clccr" (UID: "c3a5ae83-b4b8-47c6-8757-d1212f5ad638") : secret "image-registry-tls" not found Jul 25 01:11:35.665300 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:35.665125 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zc4st"] Jul 25 01:11:35.669873 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:35.669848 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8916f19_f4e7_4bbd_a0cb_49150afe5542.slice/crio-297322ce306498ddfc08dfc651853936d90fe47ee30a0fdca69eebf337f538e2 WatchSource:0}: Error finding container 297322ce306498ddfc08dfc651853936d90fe47ee30a0fdca69eebf337f538e2: Status 404 returned error can't find the container with id 297322ce306498ddfc08dfc651853936d90fe47ee30a0fdca69eebf337f538e2 Jul 25 01:11:36.098831 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.098742 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-f4cl5"] Jul 25 01:11:36.102216 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.102194 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-f4cl5" Jul 25 01:11:36.104335 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.104315 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 25 01:11:36.104882 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.104862 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 25 01:11:36.104882 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.104878 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rkv6j\"" Jul 25 01:11:36.105008 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.104910 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 25 01:11:36.105008 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.104947 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 25 01:11:36.108691 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.108669 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-f4cl5"] Jul 25 01:11:36.124151 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.124104 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:36.190034 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.190010 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:36.195772 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.195748 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-slzcl"] Jul 25 01:11:36.198991 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.198975 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-slzcl" Jul 25 01:11:36.201248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.201229 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 25 01:11:36.201344 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.201301 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 25 01:11:36.201406 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.201232 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4mg4q\"" Jul 25 01:11:36.208736 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.208715 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-slzcl"] Jul 25 01:11:36.246651 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.246624 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:36.269922 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.269897 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/631ee48a-91d4-42ef-87b3-34fd25394fab-signing-key\") pod \"service-ca-658d7b7c78-f4cl5\" (UID: \"631ee48a-91d4-42ef-87b3-34fd25394fab\") " pod="openshift-service-ca/service-ca-658d7b7c78-f4cl5" Jul 25 01:11:36.270031 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.269969 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/631ee48a-91d4-42ef-87b3-34fd25394fab-signing-cabundle\") pod \"service-ca-658d7b7c78-f4cl5\" (UID: \"631ee48a-91d4-42ef-87b3-34fd25394fab\") " pod="openshift-service-ca/service-ca-658d7b7c78-f4cl5" Jul 25 01:11:36.270031 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.270000 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s98mr\" (UniqueName: \"kubernetes.io/projected/631ee48a-91d4-42ef-87b3-34fd25394fab-kube-api-access-s98mr\") pod \"service-ca-658d7b7c78-f4cl5\" (UID: \"631ee48a-91d4-42ef-87b3-34fd25394fab\") " pod="openshift-service-ca/service-ca-658d7b7c78-f4cl5" Jul 25 01:11:36.371334 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.371263 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/631ee48a-91d4-42ef-87b3-34fd25394fab-signing-cabundle\") pod \"service-ca-658d7b7c78-f4cl5\" (UID: \"631ee48a-91d4-42ef-87b3-34fd25394fab\") " pod="openshift-service-ca/service-ca-658d7b7c78-f4cl5" Jul 25 01:11:36.371334 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.371301 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s98mr\" (UniqueName: \"kubernetes.io/projected/631ee48a-91d4-42ef-87b3-34fd25394fab-kube-api-access-s98mr\") pod \"service-ca-658d7b7c78-f4cl5\" (UID: \"631ee48a-91d4-42ef-87b3-34fd25394fab\") " pod="openshift-service-ca/service-ca-658d7b7c78-f4cl5" Jul 25 01:11:36.371334 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.371324 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5s6r\" (UniqueName: \"kubernetes.io/projected/bf1746bb-0160-47ac-81d2-4ebf692c8db1-kube-api-access-k5s6r\") pod \"migrator-6865c9967-slzcl\" (UID: \"bf1746bb-0160-47ac-81d2-4ebf692c8db1\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-slzcl" Jul 25 01:11:36.371611 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.371414 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/631ee48a-91d4-42ef-87b3-34fd25394fab-signing-key\") pod \"service-ca-658d7b7c78-f4cl5\" (UID: \"631ee48a-91d4-42ef-87b3-34fd25394fab\") " pod="openshift-service-ca/service-ca-658d7b7c78-f4cl5" Jul 25 01:11:36.372029 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.372002 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/631ee48a-91d4-42ef-87b3-34fd25394fab-signing-cabundle\") pod \"service-ca-658d7b7c78-f4cl5\" (UID: \"631ee48a-91d4-42ef-87b3-34fd25394fab\") " pod="openshift-service-ca/service-ca-658d7b7c78-f4cl5" Jul 25 01:11:36.375965 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.375939 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/631ee48a-91d4-42ef-87b3-34fd25394fab-signing-key\") pod \"service-ca-658d7b7c78-f4cl5\" (UID: \"631ee48a-91d4-42ef-87b3-34fd25394fab\") " pod="openshift-service-ca/service-ca-658d7b7c78-f4cl5" Jul 25 01:11:36.379420 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.379396 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s98mr\" (UniqueName: \"kubernetes.io/projected/631ee48a-91d4-42ef-87b3-34fd25394fab-kube-api-access-s98mr\") pod \"service-ca-658d7b7c78-f4cl5\" (UID: \"631ee48a-91d4-42ef-87b3-34fd25394fab\") " pod="openshift-service-ca/service-ca-658d7b7c78-f4cl5" Jul 25 01:11:36.413349 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.413325 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-f4cl5" Jul 25 01:11:36.475441 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.472789 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5s6r\" (UniqueName: \"kubernetes.io/projected/bf1746bb-0160-47ac-81d2-4ebf692c8db1-kube-api-access-k5s6r\") pod \"migrator-6865c9967-slzcl\" (UID: \"bf1746bb-0160-47ac-81d2-4ebf692c8db1\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-slzcl" Jul 25 01:11:36.482919 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.482861 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5s6r\" (UniqueName: \"kubernetes.io/projected/bf1746bb-0160-47ac-81d2-4ebf692c8db1-kube-api-access-k5s6r\") pod \"migrator-6865c9967-slzcl\" (UID: \"bf1746bb-0160-47ac-81d2-4ebf692c8db1\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-slzcl" Jul 25 01:11:36.511323 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.511295 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-slzcl" Jul 25 01:11:36.543285 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.543254 2554 generic.go:356] "Generic (PLEG): container finished" podID="187c72b1-1094-47c9-bb23-cbb30ab6ca4c" containerID="392d28380336d45733bc59bc31dc9a4ee33cf22ebaea3b19c7c32b9b77dc5105" exitCode=0 Jul 25 01:11:36.543424 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.543378 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhsbs" event={"ID":"187c72b1-1094-47c9-bb23-cbb30ab6ca4c","Type":"ContainerDied","Data":"392d28380336d45733bc59bc31dc9a4ee33cf22ebaea3b19c7c32b9b77dc5105"} Jul 25 01:11:36.545954 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.545927 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zc4st" event={"ID":"a8916f19-f4e7-4bbd-a0cb-49150afe5542","Type":"ContainerStarted","Data":"297322ce306498ddfc08dfc651853936d90fe47ee30a0fdca69eebf337f538e2"} Jul 25 01:11:36.569969 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.569941 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-f4cl5"] Jul 25 01:11:36.575872 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:36.575836 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631ee48a_91d4_42ef_87b3_34fd25394fab.slice/crio-986a5d13eba1f43475f7a9a7ef547605f6e86a620781127d43b3d21fc797519a WatchSource:0}: Error finding container 986a5d13eba1f43475f7a9a7ef547605f6e86a620781127d43b3d21fc797519a: Status 404 returned error can't find the container with id 986a5d13eba1f43475f7a9a7ef547605f6e86a620781127d43b3d21fc797519a Jul 25 01:11:36.675940 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.675746 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-slzcl"] Jul 25 01:11:36.679649 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:36.679620 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1746bb_0160_47ac_81d2_4ebf692c8db1.slice/crio-7e962c692511052baa728b7ea177ec6a8f664bc5a1373f3bf3af2c14bd63ff7a WatchSource:0}: Error finding container 7e962c692511052baa728b7ea177ec6a8f664bc5a1373f3bf3af2c14bd63ff7a: Status 404 returned error can't find the container with id 7e962c692511052baa728b7ea177ec6a8f664bc5a1373f3bf3af2c14bd63ff7a Jul 25 01:11:36.943401 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:36.943319 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:37.483092 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:37.482279 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f47ed40-22e9-42da-8c7a-236132c635f2-metrics-tls\") pod \"dns-default-l9pbg\" (UID: \"8f47ed40-22e9-42da-8c7a-236132c635f2\") " pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:37.483092 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:37.482348 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:37.483092 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:37.482547 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 25 01:11:37.483092 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:37.482561 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69dbd56bdd-clccr: secret "image-registry-tls" not found Jul 25 01:11:37.483092 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:37.482618 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls podName:c3a5ae83-b4b8-47c6-8757-d1212f5ad638 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:41.48259905 +0000 UTC m=+42.604084881 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls") pod "image-registry-69dbd56bdd-clccr" (UID: "c3a5ae83-b4b8-47c6-8757-d1212f5ad638") : secret "image-registry-tls" not found Jul 25 01:11:37.483092 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:37.483026 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 25 01:11:37.483092 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:37.483066 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f47ed40-22e9-42da-8c7a-236132c635f2-metrics-tls podName:8f47ed40-22e9-42da-8c7a-236132c635f2 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:41.483054757 +0000 UTC m=+42.604540584 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f47ed40-22e9-42da-8c7a-236132c635f2-metrics-tls") pod "dns-default-l9pbg" (UID: "8f47ed40-22e9-42da-8c7a-236132c635f2") : secret "dns-default-metrics-tls" not found Jul 25 01:11:37.550544 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:37.550505 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-f4cl5" event={"ID":"631ee48a-91d4-42ef-87b3-34fd25394fab","Type":"ContainerStarted","Data":"986a5d13eba1f43475f7a9a7ef547605f6e86a620781127d43b3d21fc797519a"} Jul 25 01:11:37.554714 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:37.554685 2554 generic.go:356] "Generic (PLEG): container finished" podID="187c72b1-1094-47c9-bb23-cbb30ab6ca4c" containerID="2910c305e5e11353776e110d9d1f2570f76ffd61d7f55bf33be8278622f64ba5" exitCode=0 Jul 25 01:11:37.554857 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:37.554771 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhsbs" event={"ID":"187c72b1-1094-47c9-bb23-cbb30ab6ca4c","Type":"ContainerDied","Data":"2910c305e5e11353776e110d9d1f2570f76ffd61d7f55bf33be8278622f64ba5"} Jul 25 01:11:37.556388 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:37.556361 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-slzcl" event={"ID":"bf1746bb-0160-47ac-81d2-4ebf692c8db1","Type":"ContainerStarted","Data":"7e962c692511052baa728b7ea177ec6a8f664bc5a1373f3bf3af2c14bd63ff7a"} Jul 25 01:11:38.258286 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:38.258231 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:38.312744 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:38.312708 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:40.165338 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:40.165313 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:40.564647 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:40.564562 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-f4cl5" event={"ID":"631ee48a-91d4-42ef-87b3-34fd25394fab","Type":"ContainerStarted","Data":"5ac56485e9c079b6e1719c69a81c37ab07307ca64e389ead221278959b0f33ae"} Jul 25 01:11:40.568116 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:40.568083 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhsbs" event={"ID":"187c72b1-1094-47c9-bb23-cbb30ab6ca4c","Type":"ContainerStarted","Data":"d766a53874fa485cce47bd7931891d8681e08a7275090b9e2b8cc4d74c0c9e36"} Jul 25 01:11:40.569856 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:40.569827 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-slzcl" event={"ID":"bf1746bb-0160-47ac-81d2-4ebf692c8db1","Type":"ContainerStarted","Data":"e0d79b27b4dec691e2c756bf1a67900b84386e5d2e5ca0fe34032bb95144b454"} Jul 25 01:11:40.569979 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:40.569869 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-slzcl" event={"ID":"bf1746bb-0160-47ac-81d2-4ebf692c8db1","Type":"ContainerStarted","Data":"83e23a8c958540a99023726dabcef3093016c012e5bbebc7d87fc452b9ea6160"} Jul 25 01:11:40.571215 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:40.571191 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zc4st" event={"ID":"a8916f19-f4e7-4bbd-a0cb-49150afe5542","Type":"ContainerStarted","Data":"3ccf6e5d3d83f805d476b9312e9f65e1a2623794388fde3484b0d573473bd51a"} Jul 25 01:11:40.580569 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:40.580518 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-f4cl5" podStartSLOduration=1.5752806879999999 podStartE2EDuration="4.580496955s" totalImagesPullingTime="3.005216267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:11:40.580388406 +0000 UTC m=+41.701874268" watchObservedRunningTime="2026-07-25 01:11:40.580496955 +0000 UTC m=+41.701982803" Jul 25 01:11:40.596133 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:40.596091 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zc4st" podStartSLOduration=3.689678849 podStartE2EDuration="7.596079199s" totalImagesPullingTime="3.90640035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:11:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:11:40.595338516 +0000 UTC m=+41.716824368" watchObservedRunningTime="2026-07-25 01:11:40.596079199 +0000 UTC m=+41.717565063" Jul 25 01:11:40.613628 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:40.613589 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-slzcl" podStartSLOduration=1.7215703279999999 podStartE2EDuration="4.613576216s" totalImagesPullingTime="2.892005888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:11:40.612561883 +0000 UTC m=+41.734047732" watchObservedRunningTime="2026-07-25 01:11:40.613576216 +0000 UTC m=+41.735062104" Jul 25 01:11:40.633649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:40.633598 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xhsbs" podStartSLOduration=11.535773159 podStartE2EDuration="41.633586963s" totalImagesPullingTime="30.097813804s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:10:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:11:40.632759711 +0000 UTC m=+41.754245561" watchObservedRunningTime="2026-07-25 01:11:40.633586963 +0000 UTC m=+41.755072814" Jul 25 01:11:40.898865 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:40.898765 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:40.999484 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:40.999439 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:41.100953 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:41.100915 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:41.201691 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:41.201649 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:41.281689 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:41.281659 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:41.517757 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:41.517683 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:41.517901 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:41.517794 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f47ed40-22e9-42da-8c7a-236132c635f2-metrics-tls\") pod \"dns-default-l9pbg\" (UID: \"8f47ed40-22e9-42da-8c7a-236132c635f2\") " pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:41.517901 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:41.517869 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 25 01:11:41.517901 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:41.517885 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69dbd56bdd-clccr: secret "image-registry-tls" not found Jul 25 01:11:41.517998 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:41.517901 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 25 01:11:41.517998 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:41.517961 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls podName:c3a5ae83-b4b8-47c6-8757-d1212f5ad638 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:49.517945285 +0000 UTC m=+50.639431113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls") pod "image-registry-69dbd56bdd-clccr" (UID: "c3a5ae83-b4b8-47c6-8757-d1212f5ad638") : secret "image-registry-tls" not found Jul 25 01:11:41.517998 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:11:41.517977 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f47ed40-22e9-42da-8c7a-236132c635f2-metrics-tls podName:8f47ed40-22e9-42da-8c7a-236132c635f2 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:49.517971369 +0000 UTC m=+50.639457197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f47ed40-22e9-42da-8c7a-236132c635f2-metrics-tls") pod "dns-default-l9pbg" (UID: "8f47ed40-22e9-42da-8c7a-236132c635f2") : secret "dns-default-metrics-tls" not found Jul 25 01:11:42.385204 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:42.385174 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:42.592281 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:42.592252 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:42.794460 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:42.794432 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:42.985428 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:42.985398 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:43.203742 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:43.203718 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" limit=1024 Jul 25 01:11:49.573536 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:49.573504 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f47ed40-22e9-42da-8c7a-236132c635f2-metrics-tls\") pod \"dns-default-l9pbg\" (UID: \"8f47ed40-22e9-42da-8c7a-236132c635f2\") " pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:49.574047 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:49.573547 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:49.576613 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:49.576586 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f47ed40-22e9-42da-8c7a-236132c635f2-metrics-tls\") pod \"dns-default-l9pbg\" (UID: \"8f47ed40-22e9-42da-8c7a-236132c635f2\") " pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:49.576719 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:49.576696 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls\") pod \"image-registry-69dbd56bdd-clccr\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:49.677778 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:49.677751 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:49.687054 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:49.687036 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:49.838728 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:49.838660 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69dbd56bdd-clccr"] Jul 25 01:11:49.841648 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:49.841624 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a5ae83_b4b8_47c6_8757_d1212f5ad638.slice/crio-c3aef7fc4e5e78cf09ab9d6f9113a78fb509078c27ba51c303dc3740eb861187 WatchSource:0}: Error finding container c3aef7fc4e5e78cf09ab9d6f9113a78fb509078c27ba51c303dc3740eb861187: Status 404 returned error can't find the container with id c3aef7fc4e5e78cf09ab9d6f9113a78fb509078c27ba51c303dc3740eb861187 Jul 25 01:11:49.853549 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:49.853516 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l9pbg"] Jul 25 01:11:49.858749 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:49.858730 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f47ed40_22e9_42da_8c7a_236132c635f2.slice/crio-4b7d66d76e41716cf39b4ec7be82157799e231fc86e07882b770ea97f376feb4 WatchSource:0}: Error finding container 4b7d66d76e41716cf39b4ec7be82157799e231fc86e07882b770ea97f376feb4: Status 404 returned error can't find the container with id 4b7d66d76e41716cf39b4ec7be82157799e231fc86e07882b770ea97f376feb4 Jul 25 01:11:50.596700 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:50.596458 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" event={"ID":"c3a5ae83-b4b8-47c6-8757-d1212f5ad638","Type":"ContainerStarted","Data":"b1f39897a02bf2eaa83358d47cc3cfe0fcd44389e362006c2cabe8f3e13ebe5e"} Jul 25 01:11:50.596700 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:50.596512 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" event={"ID":"c3a5ae83-b4b8-47c6-8757-d1212f5ad638","Type":"ContainerStarted","Data":"c3aef7fc4e5e78cf09ab9d6f9113a78fb509078c27ba51c303dc3740eb861187"} Jul 25 01:11:50.596700 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:50.596584 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:11:50.597773 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:50.597752 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l9pbg" event={"ID":"8f47ed40-22e9-42da-8c7a-236132c635f2","Type":"ContainerStarted","Data":"4b7d66d76e41716cf39b4ec7be82157799e231fc86e07882b770ea97f376feb4"} Jul 25 01:11:50.616344 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:50.616293 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" podStartSLOduration=51.616279131 podStartE2EDuration="51.616279131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:11:50.614639305 +0000 UTC m=+51.736125156" watchObservedRunningTime="2026-07-25 01:11:50.616279131 +0000 UTC m=+51.737764980" Jul 25 01:11:52.607699 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:52.607617 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l9pbg" event={"ID":"8f47ed40-22e9-42da-8c7a-236132c635f2","Type":"ContainerStarted","Data":"d13258cec516201f36e8c28debf1692742fcc1175bc22dd5596ec51bd3e921c1"} Jul 25 01:11:52.607699 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:52.607668 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l9pbg" event={"ID":"8f47ed40-22e9-42da-8c7a-236132c635f2","Type":"ContainerStarted","Data":"cf5910b9f9c026f4342304887b10a8d0cb2d42c9f9f2c7c5f25e6f1e26d69b3b"} Jul 25 01:11:52.608154 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:52.607708 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-l9pbg" Jul 25 01:11:52.626117 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:52.626068 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-l9pbg" podStartSLOduration=17.481318659 podStartE2EDuration="19.626053824s" totalImagesPullingTime="2.144735165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:11:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:11:52.624970463 +0000 UTC m=+53.746456313" watchObservedRunningTime="2026-07-25 01:11:52.626053824 +0000 UTC m=+53.747539678" Jul 25 01:11:57.245268 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.245234 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6cdd8d58d5-6v7sn"] Jul 25 01:11:57.247773 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.247754 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:57.250446 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.250420 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 25 01:11:57.250446 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.250439 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 25 01:11:57.250764 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.250420 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 25 01:11:57.250764 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.250717 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 25 01:11:57.250958 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.250942 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 25 01:11:57.251170 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.251154 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 25 01:11:57.251423 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.251409 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xkbxp\"" Jul 25 01:11:57.260005 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.259985 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-744df4dd6d-lbpn8"] Jul 25 01:11:57.261629 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.261610 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744df4dd6d-lbpn8" Jul 25 01:11:57.263834 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.263783 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 25 01:11:57.264427 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.264410 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 25 01:11:57.264500 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.264458 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 25 01:11:57.265652 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.265637 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 25 01:11:57.271593 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.271575 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6cdd8d58d5-6v7sn"] Jul 25 01:11:57.282165 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.282146 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-744df4dd6d-lbpn8"] Jul 25 01:11:57.282889 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.282872 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f944c58bb-l9hwr"] Jul 25 01:11:57.284676 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.284659 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f944c58bb-l9hwr" Jul 25 01:11:57.286946 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.286929 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 25 01:11:57.287344 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.287328 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-hmq8d\"" Jul 25 01:11:57.292751 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.292733 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69dbd56bdd-clccr"] Jul 25 01:11:57.299468 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.299443 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f944c58bb-l9hwr"] Jul 25 01:11:57.340847 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.340818 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-fqrtv"] Jul 25 01:11:57.342630 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.342613 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fqrtv" Jul 25 01:11:57.344930 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.344911 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 25 01:11:57.345291 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.345273 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 25 01:11:57.345950 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.345921 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 25 01:11:57.346038 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.345965 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 25 01:11:57.346038 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.346021 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-k6w7z\"" Jul 25 01:11:57.355271 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.355252 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-lnzxj"] Jul 25 01:11:57.356972 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.356957 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d99ff7995-kcnlv"] Jul 25 01:11:57.357076 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.357061 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.358661 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.358644 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.359176 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.359121 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 25 01:11:57.360146 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.360009 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 25 01:11:57.360146 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.360030 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 25 01:11:57.360312 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.360223 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 25 01:11:57.360435 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.360411 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-fqrtv"] Jul 25 01:11:57.360543 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.360453 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vs8d9\"" Jul 25 01:11:57.367304 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.367284 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 25 01:11:57.371844 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.371825 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-lnzxj"] Jul 25 01:11:57.379036 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.379017 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d99ff7995-kcnlv"] Jul 25 01:11:57.427886 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.427855 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/207f125a-e453-4137-95d2-29c48898e3bf-stats-auth\") pod \"router-default-6cdd8d58d5-6v7sn\" (UID: \"207f125a-e453-4137-95d2-29c48898e3bf\") " pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:57.427886 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.427888 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207f125a-e453-4137-95d2-29c48898e3bf-service-ca-bundle\") pod \"router-default-6cdd8d58d5-6v7sn\" (UID: \"207f125a-e453-4137-95d2-29c48898e3bf\") " pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:57.428080 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.427916 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207f125a-e453-4137-95d2-29c48898e3bf-metrics-certs\") pod \"router-default-6cdd8d58d5-6v7sn\" (UID: \"207f125a-e453-4137-95d2-29c48898e3bf\") " pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:57.428080 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.428010 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf903392-e0b9-4411-a0ff-236c29df8d0f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f944c58bb-l9hwr\" (UID: \"bf903392-e0b9-4411-a0ff-236c29df8d0f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f944c58bb-l9hwr" Jul 25 01:11:57.428080 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.428041 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd2ch\" (UniqueName: \"kubernetes.io/projected/bf903392-e0b9-4411-a0ff-236c29df8d0f-kube-api-access-kd2ch\") pod \"managed-serviceaccount-addon-agent-f944c58bb-l9hwr\" (UID: \"bf903392-e0b9-4411-a0ff-236c29df8d0f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f944c58bb-l9hwr" Jul 25 01:11:57.428080 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.428069 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/207f125a-e453-4137-95d2-29c48898e3bf-default-certificate\") pod \"router-default-6cdd8d58d5-6v7sn\" (UID: \"207f125a-e453-4137-95d2-29c48898e3bf\") " pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:57.428227 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.428088 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4j8x\" (UniqueName: \"kubernetes.io/projected/fc55f5c8-abc9-4f86-8362-1690734dd7a6-kube-api-access-d4j8x\") pod \"klusterlet-addon-workmgr-744df4dd6d-lbpn8\" (UID: \"fc55f5c8-abc9-4f86-8362-1690734dd7a6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744df4dd6d-lbpn8" Jul 25 01:11:57.428227 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.428105 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc55f5c8-abc9-4f86-8362-1690734dd7a6-tmp\") pod \"klusterlet-addon-workmgr-744df4dd6d-lbpn8\" (UID: \"fc55f5c8-abc9-4f86-8362-1690734dd7a6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744df4dd6d-lbpn8" Jul 25 01:11:57.428227 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.428129 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhjt\" (UniqueName: \"kubernetes.io/projected/207f125a-e453-4137-95d2-29c48898e3bf-kube-api-access-pjhjt\") pod \"router-default-6cdd8d58d5-6v7sn\" (UID: \"207f125a-e453-4137-95d2-29c48898e3bf\") " pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:57.428227 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.428163 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fc55f5c8-abc9-4f86-8362-1690734dd7a6-klusterlet-config\") pod \"klusterlet-addon-workmgr-744df4dd6d-lbpn8\" (UID: \"fc55f5c8-abc9-4f86-8362-1690734dd7a6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744df4dd6d-lbpn8" Jul 25 01:11:57.528663 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.528580 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4j8x\" (UniqueName: \"kubernetes.io/projected/fc55f5c8-abc9-4f86-8362-1690734dd7a6-kube-api-access-d4j8x\") pod \"klusterlet-addon-workmgr-744df4dd6d-lbpn8\" (UID: \"fc55f5c8-abc9-4f86-8362-1690734dd7a6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744df4dd6d-lbpn8" Jul 25 01:11:57.528663 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.528621 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc55f5c8-abc9-4f86-8362-1690734dd7a6-tmp\") pod \"klusterlet-addon-workmgr-744df4dd6d-lbpn8\" (UID: \"fc55f5c8-abc9-4f86-8362-1690734dd7a6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744df4dd6d-lbpn8" Jul 25 01:11:57.528663 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.528642 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f6485a23-898b-4080-bb1c-0f431957ff23-snapshots\") pod \"insights-operator-5885fdb759-lnzxj\" (UID: \"f6485a23-898b-4080-bb1c-0f431957ff23\") " pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.528663 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.528661 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0d77b32a-87c3-4ea9-9d10-5eb70418a914-registry-certificates\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.529000 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.528732 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhjt\" (UniqueName: \"kubernetes.io/projected/207f125a-e453-4137-95d2-29c48898e3bf-kube-api-access-pjhjt\") pod \"router-default-6cdd8d58d5-6v7sn\" (UID: \"207f125a-e453-4137-95d2-29c48898e3bf\") " pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:57.529000 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.528766 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fc55f5c8-abc9-4f86-8362-1690734dd7a6-klusterlet-config\") pod \"klusterlet-addon-workmgr-744df4dd6d-lbpn8\" (UID: \"fc55f5c8-abc9-4f86-8362-1690734dd7a6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744df4dd6d-lbpn8" Jul 25 01:11:57.529000 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.528786 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6485a23-898b-4080-bb1c-0f431957ff23-service-ca-bundle\") pod \"insights-operator-5885fdb759-lnzxj\" (UID: \"f6485a23-898b-4080-bb1c-0f431957ff23\") " pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.529000 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.528838 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0d77b32a-87c3-4ea9-9d10-5eb70418a914-installation-pull-secrets\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.529000 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.528891 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0d77b32a-87c3-4ea9-9d10-5eb70418a914-trusted-ca\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.529000 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.528929 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d77b32a-87c3-4ea9-9d10-5eb70418a914-bound-sa-token\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.529000 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.528966 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/207f125a-e453-4137-95d2-29c48898e3bf-stats-auth\") pod \"router-default-6cdd8d58d5-6v7sn\" (UID: \"207f125a-e453-4137-95d2-29c48898e3bf\") " pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:57.529000 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.528990 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc55f5c8-abc9-4f86-8362-1690734dd7a6-tmp\") pod \"klusterlet-addon-workmgr-744df4dd6d-lbpn8\" (UID: \"fc55f5c8-abc9-4f86-8362-1690734dd7a6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744df4dd6d-lbpn8" Jul 25 01:11:57.529347 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.528989 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0d77b32a-87c3-4ea9-9d10-5eb70418a914-image-registry-private-configuration\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.529347 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.529037 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207f125a-e453-4137-95d2-29c48898e3bf-service-ca-bundle\") pod \"router-default-6cdd8d58d5-6v7sn\" (UID: \"207f125a-e453-4137-95d2-29c48898e3bf\") " pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:57.529347 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.529065 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0d77b32a-87c3-4ea9-9d10-5eb70418a914-registry-tls\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.529347 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.529090 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6485a23-898b-4080-bb1c-0f431957ff23-serving-cert\") pod \"insights-operator-5885fdb759-lnzxj\" (UID: \"f6485a23-898b-4080-bb1c-0f431957ff23\") " pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.529347 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.529204 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6485a23-898b-4080-bb1c-0f431957ff23-tmp\") pod \"insights-operator-5885fdb759-lnzxj\" (UID: \"f6485a23-898b-4080-bb1c-0f431957ff23\") " pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.529347 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.529238 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207f125a-e453-4137-95d2-29c48898e3bf-metrics-certs\") pod \"router-default-6cdd8d58d5-6v7sn\" (UID: \"207f125a-e453-4137-95d2-29c48898e3bf\") " pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:57.529347 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.529265 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q26h\" (UniqueName: \"kubernetes.io/projected/f6485a23-898b-4080-bb1c-0f431957ff23-kube-api-access-5q26h\") pod \"insights-operator-5885fdb759-lnzxj\" (UID: \"f6485a23-898b-4080-bb1c-0f431957ff23\") " pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.529682 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.529412 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42cfeae6-5082-45ae-a2e8-2325c0ca5ce5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-fqrtv\" (UID: \"42cfeae6-5082-45ae-a2e8-2325c0ca5ce5\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fqrtv" Jul 25 01:11:57.529682 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.529454 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0d77b32a-87c3-4ea9-9d10-5eb70418a914-ca-trust-extracted\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.529682 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.529488 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6485a23-898b-4080-bb1c-0f431957ff23-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-lnzxj\" (UID: \"f6485a23-898b-4080-bb1c-0f431957ff23\") " pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.529682 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.529549 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/42cfeae6-5082-45ae-a2e8-2325c0ca5ce5-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-fqrtv\" (UID: \"42cfeae6-5082-45ae-a2e8-2325c0ca5ce5\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fqrtv" Jul 25 01:11:57.529682 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.529603 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf903392-e0b9-4411-a0ff-236c29df8d0f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f944c58bb-l9hwr\" (UID: \"bf903392-e0b9-4411-a0ff-236c29df8d0f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f944c58bb-l9hwr" Jul 25 01:11:57.529682 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.529631 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6klw7\" (UniqueName: \"kubernetes.io/projected/42cfeae6-5082-45ae-a2e8-2325c0ca5ce5-kube-api-access-6klw7\") pod \"cluster-monitoring-operator-548b489cb8-fqrtv\" (UID: \"42cfeae6-5082-45ae-a2e8-2325c0ca5ce5\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fqrtv" Jul 25 01:11:57.529682 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.529661 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207f125a-e453-4137-95d2-29c48898e3bf-service-ca-bundle\") pod \"router-default-6cdd8d58d5-6v7sn\" (UID: \"207f125a-e453-4137-95d2-29c48898e3bf\") " pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:57.530015 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.529662 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd2ch\" (UniqueName: \"kubernetes.io/projected/bf903392-e0b9-4411-a0ff-236c29df8d0f-kube-api-access-kd2ch\") pod \"managed-serviceaccount-addon-agent-f944c58bb-l9hwr\" (UID: \"bf903392-e0b9-4411-a0ff-236c29df8d0f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f944c58bb-l9hwr" Jul 25 01:11:57.530015 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.529735 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/207f125a-e453-4137-95d2-29c48898e3bf-default-certificate\") pod \"router-default-6cdd8d58d5-6v7sn\" (UID: \"207f125a-e453-4137-95d2-29c48898e3bf\") " pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:57.530015 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.529764 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcn2b\" (UniqueName: \"kubernetes.io/projected/0d77b32a-87c3-4ea9-9d10-5eb70418a914-kube-api-access-qcn2b\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.531485 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.531465 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fc55f5c8-abc9-4f86-8362-1690734dd7a6-klusterlet-config\") pod \"klusterlet-addon-workmgr-744df4dd6d-lbpn8\" (UID: \"fc55f5c8-abc9-4f86-8362-1690734dd7a6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744df4dd6d-lbpn8" Jul 25 01:11:57.531902 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.531876 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/207f125a-e453-4137-95d2-29c48898e3bf-stats-auth\") pod \"router-default-6cdd8d58d5-6v7sn\" (UID: \"207f125a-e453-4137-95d2-29c48898e3bf\") " pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:57.531988 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.531947 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207f125a-e453-4137-95d2-29c48898e3bf-metrics-certs\") pod \"router-default-6cdd8d58d5-6v7sn\" (UID: \"207f125a-e453-4137-95d2-29c48898e3bf\") " pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:57.532082 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.532057 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/207f125a-e453-4137-95d2-29c48898e3bf-default-certificate\") pod \"router-default-6cdd8d58d5-6v7sn\" (UID: \"207f125a-e453-4137-95d2-29c48898e3bf\") " pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:57.532326 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.532305 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf903392-e0b9-4411-a0ff-236c29df8d0f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f944c58bb-l9hwr\" (UID: \"bf903392-e0b9-4411-a0ff-236c29df8d0f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f944c58bb-l9hwr" Jul 25 01:11:57.537832 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.537789 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4j8x\" (UniqueName: \"kubernetes.io/projected/fc55f5c8-abc9-4f86-8362-1690734dd7a6-kube-api-access-d4j8x\") pod \"klusterlet-addon-workmgr-744df4dd6d-lbpn8\" (UID: \"fc55f5c8-abc9-4f86-8362-1690734dd7a6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744df4dd6d-lbpn8" Jul 25 01:11:57.538323 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.538300 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhjt\" (UniqueName: \"kubernetes.io/projected/207f125a-e453-4137-95d2-29c48898e3bf-kube-api-access-pjhjt\") pod \"router-default-6cdd8d58d5-6v7sn\" (UID: \"207f125a-e453-4137-95d2-29c48898e3bf\") " pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:57.539506 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.539480 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd2ch\" (UniqueName: \"kubernetes.io/projected/bf903392-e0b9-4411-a0ff-236c29df8d0f-kube-api-access-kd2ch\") pod \"managed-serviceaccount-addon-agent-f944c58bb-l9hwr\" (UID: \"bf903392-e0b9-4411-a0ff-236c29df8d0f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f944c58bb-l9hwr" Jul 25 01:11:57.556330 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.556313 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:57.570854 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.570834 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744df4dd6d-lbpn8" Jul 25 01:11:57.605443 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.605387 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f944c58bb-l9hwr" Jul 25 01:11:57.630629 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.630564 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6485a23-898b-4080-bb1c-0f431957ff23-service-ca-bundle\") pod \"insights-operator-5885fdb759-lnzxj\" (UID: \"f6485a23-898b-4080-bb1c-0f431957ff23\") " pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.630629 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.630613 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0d77b32a-87c3-4ea9-9d10-5eb70418a914-installation-pull-secrets\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.630831 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.630645 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0d77b32a-87c3-4ea9-9d10-5eb70418a914-trusted-ca\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.630831 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.630669 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d77b32a-87c3-4ea9-9d10-5eb70418a914-bound-sa-token\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.630831 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.630697 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0d77b32a-87c3-4ea9-9d10-5eb70418a914-image-registry-private-configuration\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.630831 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.630729 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0d77b32a-87c3-4ea9-9d10-5eb70418a914-registry-tls\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.630831 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.630751 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6485a23-898b-4080-bb1c-0f431957ff23-serving-cert\") pod \"insights-operator-5885fdb759-lnzxj\" (UID: \"f6485a23-898b-4080-bb1c-0f431957ff23\") " pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.630831 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.630780 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6485a23-898b-4080-bb1c-0f431957ff23-tmp\") pod \"insights-operator-5885fdb759-lnzxj\" (UID: \"f6485a23-898b-4080-bb1c-0f431957ff23\") " pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.630831 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.630828 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q26h\" (UniqueName: \"kubernetes.io/projected/f6485a23-898b-4080-bb1c-0f431957ff23-kube-api-access-5q26h\") pod \"insights-operator-5885fdb759-lnzxj\" (UID: \"f6485a23-898b-4080-bb1c-0f431957ff23\") " pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.631152 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.630895 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42cfeae6-5082-45ae-a2e8-2325c0ca5ce5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-fqrtv\" (UID: \"42cfeae6-5082-45ae-a2e8-2325c0ca5ce5\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fqrtv" Jul 25 01:11:57.631152 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.630926 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0d77b32a-87c3-4ea9-9d10-5eb70418a914-ca-trust-extracted\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.631152 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.631014 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6485a23-898b-4080-bb1c-0f431957ff23-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-lnzxj\" (UID: \"f6485a23-898b-4080-bb1c-0f431957ff23\") " pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.631152 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.631059 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/42cfeae6-5082-45ae-a2e8-2325c0ca5ce5-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-fqrtv\" (UID: \"42cfeae6-5082-45ae-a2e8-2325c0ca5ce5\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fqrtv" Jul 25 01:11:57.631152 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.631111 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6klw7\" (UniqueName: \"kubernetes.io/projected/42cfeae6-5082-45ae-a2e8-2325c0ca5ce5-kube-api-access-6klw7\") pod \"cluster-monitoring-operator-548b489cb8-fqrtv\" (UID: \"42cfeae6-5082-45ae-a2e8-2325c0ca5ce5\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fqrtv" Jul 25 01:11:57.631920 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.631164 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcn2b\" (UniqueName: \"kubernetes.io/projected/0d77b32a-87c3-4ea9-9d10-5eb70418a914-kube-api-access-qcn2b\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.631920 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.631199 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f6485a23-898b-4080-bb1c-0f431957ff23-snapshots\") pod \"insights-operator-5885fdb759-lnzxj\" (UID: \"f6485a23-898b-4080-bb1c-0f431957ff23\") " pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.631920 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.631227 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0d77b32a-87c3-4ea9-9d10-5eb70418a914-registry-certificates\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.631920 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.631397 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6485a23-898b-4080-bb1c-0f431957ff23-service-ca-bundle\") pod \"insights-operator-5885fdb759-lnzxj\" (UID: \"f6485a23-898b-4080-bb1c-0f431957ff23\") " pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.632126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.632055 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0d77b32a-87c3-4ea9-9d10-5eb70418a914-registry-certificates\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.634351 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.632309 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0d77b32a-87c3-4ea9-9d10-5eb70418a914-trusted-ca\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.634351 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.632337 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6485a23-898b-4080-bb1c-0f431957ff23-tmp\") pod \"insights-operator-5885fdb759-lnzxj\" (UID: \"f6485a23-898b-4080-bb1c-0f431957ff23\") " pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.634351 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.633020 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/42cfeae6-5082-45ae-a2e8-2325c0ca5ce5-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-fqrtv\" (UID: \"42cfeae6-5082-45ae-a2e8-2325c0ca5ce5\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fqrtv" Jul 25 01:11:57.634351 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.633041 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6485a23-898b-4080-bb1c-0f431957ff23-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-lnzxj\" (UID: \"f6485a23-898b-4080-bb1c-0f431957ff23\") " pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.634351 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.633787 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f6485a23-898b-4080-bb1c-0f431957ff23-snapshots\") pod \"insights-operator-5885fdb759-lnzxj\" (UID: \"f6485a23-898b-4080-bb1c-0f431957ff23\") " pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.634753 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.634713 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0d77b32a-87c3-4ea9-9d10-5eb70418a914-ca-trust-extracted\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.636460 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.636435 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0d77b32a-87c3-4ea9-9d10-5eb70418a914-installation-pull-secrets\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.639673 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.638131 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0d77b32a-87c3-4ea9-9d10-5eb70418a914-registry-tls\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.642280 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.641488 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0d77b32a-87c3-4ea9-9d10-5eb70418a914-image-registry-private-configuration\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.642280 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.641920 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42cfeae6-5082-45ae-a2e8-2325c0ca5ce5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-fqrtv\" (UID: \"42cfeae6-5082-45ae-a2e8-2325c0ca5ce5\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fqrtv" Jul 25 01:11:57.643167 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.643125 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d77b32a-87c3-4ea9-9d10-5eb70418a914-bound-sa-token\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.643338 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.643318 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6485a23-898b-4080-bb1c-0f431957ff23-serving-cert\") pod \"insights-operator-5885fdb759-lnzxj\" (UID: \"f6485a23-898b-4080-bb1c-0f431957ff23\") " pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.644150 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.644043 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcn2b\" (UniqueName: \"kubernetes.io/projected/0d77b32a-87c3-4ea9-9d10-5eb70418a914-kube-api-access-qcn2b\") pod \"image-registry-5d99ff7995-kcnlv\" (UID: \"0d77b32a-87c3-4ea9-9d10-5eb70418a914\") " pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.645646 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.645240 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q26h\" (UniqueName: \"kubernetes.io/projected/f6485a23-898b-4080-bb1c-0f431957ff23-kube-api-access-5q26h\") pod \"insights-operator-5885fdb759-lnzxj\" (UID: \"f6485a23-898b-4080-bb1c-0f431957ff23\") " pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.646568 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.646088 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6klw7\" (UniqueName: \"kubernetes.io/projected/42cfeae6-5082-45ae-a2e8-2325c0ca5ce5-kube-api-access-6klw7\") pod \"cluster-monitoring-operator-548b489cb8-fqrtv\" (UID: \"42cfeae6-5082-45ae-a2e8-2325c0ca5ce5\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fqrtv" Jul 25 01:11:57.652231 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.652059 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fqrtv" Jul 25 01:11:57.667636 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.666571 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-lnzxj" Jul 25 01:11:57.672283 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.672102 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:57.716653 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.716565 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6cdd8d58d5-6v7sn"] Jul 25 01:11:57.744257 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.744169 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-744df4dd6d-lbpn8"] Jul 25 01:11:57.755358 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:57.755327 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc55f5c8_abc9_4f86_8362_1690734dd7a6.slice/crio-91c83c84eb339ac211152ef5e07a44b7c252c92605ac695f16a9e5306a53f0f7 WatchSource:0}: Error finding container 91c83c84eb339ac211152ef5e07a44b7c252c92605ac695f16a9e5306a53f0f7: Status 404 returned error can't find the container with id 91c83c84eb339ac211152ef5e07a44b7c252c92605ac695f16a9e5306a53f0f7 Jul 25 01:11:57.795256 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.795195 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f944c58bb-l9hwr"] Jul 25 01:11:57.827354 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.827320 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-fqrtv"] Jul 25 01:11:57.830706 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:57.830661 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42cfeae6_5082_45ae_a2e8_2325c0ca5ce5.slice/crio-261c444add9dc4cacaed7da964489842aa96bd61f5a06143994a8bb0822b8f4d WatchSource:0}: Error finding container 261c444add9dc4cacaed7da964489842aa96bd61f5a06143994a8bb0822b8f4d: Status 404 returned error can't find the container with id 261c444add9dc4cacaed7da964489842aa96bd61f5a06143994a8bb0822b8f4d Jul 25 01:11:57.848619 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.848595 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-lnzxj"] Jul 25 01:11:57.851385 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:57.851355 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6485a23_898b_4080_bb1c_0f431957ff23.slice/crio-2f772d80d4e0e156e021069f7b19270540d2e8ee314114d9dff99a4f3f0c36ff WatchSource:0}: Error finding container 2f772d80d4e0e156e021069f7b19270540d2e8ee314114d9dff99a4f3f0c36ff: Status 404 returned error can't find the container with id 2f772d80d4e0e156e021069f7b19270540d2e8ee314114d9dff99a4f3f0c36ff Jul 25 01:11:57.864583 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:57.864559 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d99ff7995-kcnlv"] Jul 25 01:11:57.868507 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:11:57.868467 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d77b32a_87c3_4ea9_9d10_5eb70418a914.slice/crio-66e431f85447b759d41797e75551a8430507374fbae873dbbc337276387af2da WatchSource:0}: Error finding container 66e431f85447b759d41797e75551a8430507374fbae873dbbc337276387af2da: Status 404 returned error can't find the container with id 66e431f85447b759d41797e75551a8430507374fbae873dbbc337276387af2da Jul 25 01:11:58.633704 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:58.633565 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f944c58bb-l9hwr" event={"ID":"bf903392-e0b9-4411-a0ff-236c29df8d0f","Type":"ContainerStarted","Data":"1ec09c6ec51598153f884c171b82abee2e400c547ef9c095983674fdefaaae3e"} Jul 25 01:11:58.636555 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:58.636356 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fqrtv" event={"ID":"42cfeae6-5082-45ae-a2e8-2325c0ca5ce5","Type":"ContainerStarted","Data":"261c444add9dc4cacaed7da964489842aa96bd61f5a06143994a8bb0822b8f4d"} Jul 25 01:11:58.638189 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:58.638118 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-lnzxj" event={"ID":"f6485a23-898b-4080-bb1c-0f431957ff23","Type":"ContainerStarted","Data":"2f772d80d4e0e156e021069f7b19270540d2e8ee314114d9dff99a4f3f0c36ff"} Jul 25 01:11:58.641628 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:58.641602 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744df4dd6d-lbpn8" event={"ID":"fc55f5c8-abc9-4f86-8362-1690734dd7a6","Type":"ContainerStarted","Data":"91c83c84eb339ac211152ef5e07a44b7c252c92605ac695f16a9e5306a53f0f7"} Jul 25 01:11:58.645115 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:58.644263 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" event={"ID":"0d77b32a-87c3-4ea9-9d10-5eb70418a914","Type":"ContainerStarted","Data":"a580739bb0411f695527d5282dc63542ad159972233626ffdca4e40b7b60f7be"} Jul 25 01:11:58.645115 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:58.644303 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" event={"ID":"0d77b32a-87c3-4ea9-9d10-5eb70418a914","Type":"ContainerStarted","Data":"66e431f85447b759d41797e75551a8430507374fbae873dbbc337276387af2da"} Jul 25 01:11:58.645115 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:58.645068 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:11:58.648328 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:58.647872 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" event={"ID":"207f125a-e453-4137-95d2-29c48898e3bf","Type":"ContainerStarted","Data":"5688d0cc6972c88453fba39ad3dca100eb4001acfdee61b5ea481c73ba4a46f4"} Jul 25 01:11:58.648328 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:58.647918 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" event={"ID":"207f125a-e453-4137-95d2-29c48898e3bf","Type":"ContainerStarted","Data":"6a54a81ded161f2c0cc4d9e24c64a9cc71ee5058f1459c6b5e7cfbbf880d5b94"} Jul 25 01:11:58.672484 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:58.671254 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" podStartSLOduration=1.6712367270000001 podStartE2EDuration="1.671236727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:11:58.670286953 +0000 UTC m=+59.791772804" watchObservedRunningTime="2026-07-25 01:11:58.671236727 +0000 UTC m=+59.792722658" Jul 25 01:11:58.689745 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:58.689703 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" podStartSLOduration=1.689684943 podStartE2EDuration="1.689684943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:11:58.688254922 +0000 UTC m=+59.809740793" watchObservedRunningTime="2026-07-25 01:11:58.689684943 +0000 UTC m=+59.811170799" Jul 25 01:11:59.543576 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:59.541284 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lph8b" Jul 25 01:11:59.556951 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:59.556926 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:59.560194 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:59.560171 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:59.655107 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:59.655041 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:11:59.657552 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:11:59.657377 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6cdd8d58d5-6v7sn" Jul 25 01:12:02.616931 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:02.616079 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l9pbg" Jul 25 01:12:02.668015 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:02.667981 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fqrtv" event={"ID":"42cfeae6-5082-45ae-a2e8-2325c0ca5ce5","Type":"ContainerStarted","Data":"5441c4b40544c5c8670e2d7dd48ee4e59515cf37cacd70e6ce555f3cd4298a66"} Jul 25 01:12:02.671604 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:02.670835 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-lnzxj" event={"ID":"f6485a23-898b-4080-bb1c-0f431957ff23","Type":"ContainerStarted","Data":"e30f98b7e33be1544c4bb1c8e460e6f1bed51898d252981061f0ee4646fd2228"} Jul 25 01:12:02.672781 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:02.672757 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f944c58bb-l9hwr" event={"ID":"bf903392-e0b9-4411-a0ff-236c29df8d0f","Type":"ContainerStarted","Data":"f5fcbffc7a076139ab3bd5ab4b18f7c0522db210f947ec50d2d9ab48ce18c95c"} Jul 25 01:12:02.728056 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:02.728010 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fqrtv" podStartSLOduration=1.471981804 podStartE2EDuration="5.727994763s" totalImagesPullingTime="4.256012959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:02.691253504 +0000 UTC m=+63.812739349" watchObservedRunningTime="2026-07-25 01:12:02.727994763 +0000 UTC m=+63.849480613" Jul 25 01:12:02.728407 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:02.728336 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f944c58bb-l9hwr" podStartSLOduration=1.426587281 podStartE2EDuration="5.728324357s" totalImagesPullingTime="4.301737076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:02.727250511 +0000 UTC m=+63.848736361" watchObservedRunningTime="2026-07-25 01:12:02.728324357 +0000 UTC m=+63.849810210" Jul 25 01:12:02.744354 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:02.744316 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-lnzxj" podStartSLOduration=1.502392782 podStartE2EDuration="5.744303994s" totalImagesPullingTime="4.241911212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:02.743275465 +0000 UTC m=+63.864761324" watchObservedRunningTime="2026-07-25 01:12:02.744303994 +0000 UTC m=+63.865789844" Jul 25 01:12:02.894929 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:02.894834 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gnkn9"] Jul 25 01:12:02.900749 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:02.900717 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:02.904396 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:02.904372 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 25 01:12:02.904514 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:02.904484 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 25 01:12:02.904595 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:02.904571 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z9n5z\"" Jul 25 01:12:02.909655 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:02.909587 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gnkn9"] Jul 25 01:12:02.991244 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:02.991207 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-data-volume\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:02.991244 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:02.991258 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:02.991471 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:02.991353 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:02.991471 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:02.991392 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-crio-socket\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:02.991572 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:02.991509 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdl2x\" (UniqueName: \"kubernetes.io/projected/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-kube-api-access-bdl2x\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:03.092355 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:03.092317 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-data-volume\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:03.092528 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:03.092364 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:03.092528 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:03.092426 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:03.092528 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:03.092458 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-crio-socket\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:03.092651 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:03.092538 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-crio-socket\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:03.092651 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:12:03.092561 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 25 01:12:03.092651 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:12:03.092610 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 25 01:12:03.092651 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:12:03.092637 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-insights-runtime-extractor-tls podName:25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b nodeName:}" failed. No retries permitted until 2026-07-25 01:12:03.592614515 +0000 UTC m=+64.714100349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnkn9" (UID: "25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b") : secret "insights-runtime-extractor-tls" not found Jul 25 01:12:03.092794 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:12:03.092665 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-kube-rbac-proxy-cm podName:25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b nodeName:}" failed. No retries permitted until 2026-07-25 01:12:03.592648556 +0000 UTC m=+64.714134393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnkn9" (UID: "25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b") : configmap "kube-rbac-proxy" not found Jul 25 01:12:03.092794 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:03.092713 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-data-volume\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:03.092794 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:03.092734 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdl2x\" (UniqueName: \"kubernetes.io/projected/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-kube-api-access-bdl2x\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:03.108999 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:03.108966 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdl2x\" (UniqueName: \"kubernetes.io/projected/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-kube-api-access-bdl2x\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:03.598374 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:03.598344 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:03.598530 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:03.598426 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:03.598530 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:12:03.598465 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 25 01:12:03.598601 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:12:03.598533 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 25 01:12:03.598601 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:12:03.598538 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-kube-rbac-proxy-cm podName:25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b nodeName:}" failed. No retries permitted until 2026-07-25 01:12:04.598519937 +0000 UTC m=+65.720005769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnkn9" (UID: "25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b") : configmap "kube-rbac-proxy" not found Jul 25 01:12:03.598601 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:12:03.598574 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-insights-runtime-extractor-tls podName:25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b nodeName:}" failed. No retries permitted until 2026-07-25 01:12:04.598562677 +0000 UTC m=+65.720048505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnkn9" (UID: "25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b") : secret "insights-runtime-extractor-tls" not found Jul 25 01:12:03.909503 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:03.909396 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l9pbg_8f47ed40-22e9-42da-8c7a-236132c635f2/dns/0.log" Jul 25 01:12:04.111872 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:04.111842 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l9pbg_8f47ed40-22e9-42da-8c7a-236132c635f2/kube-rbac-proxy/0.log" Jul 25 01:12:04.607122 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:04.607087 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:04.607310 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:04.607142 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:04.607310 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:12:04.607238 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 25 01:12:04.607310 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:12:04.607240 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 25 01:12:04.607310 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:12:04.607303 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-insights-runtime-extractor-tls podName:25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b nodeName:}" failed. No retries permitted until 2026-07-25 01:12:06.607287362 +0000 UTC m=+67.728773190 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnkn9" (UID: "25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b") : secret "insights-runtime-extractor-tls" not found Jul 25 01:12:04.607462 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:12:04.607316 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-kube-rbac-proxy-cm podName:25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b nodeName:}" failed. No retries permitted until 2026-07-25 01:12:06.60731046 +0000 UTC m=+67.728796288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnkn9" (UID: "25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b") : configmap "kube-rbac-proxy" not found Jul 25 01:12:04.679649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:04.679616 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744df4dd6d-lbpn8" event={"ID":"fc55f5c8-abc9-4f86-8362-1690734dd7a6","Type":"ContainerStarted","Data":"cbcebfdb11abdd3864d0947feb19c4136cd44662e91358b788bf0f123ab38788"} Jul 25 01:12:04.679859 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:04.679838 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744df4dd6d-lbpn8" Jul 25 01:12:04.681592 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:04.681571 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744df4dd6d-lbpn8" Jul 25 01:12:04.694055 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:04.694010 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744df4dd6d-lbpn8" podStartSLOduration=1.7817794359999999 podStartE2EDuration="7.693997077s" totalImagesPullingTime="5.912217641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:04.692466052 +0000 UTC m=+65.813951924" watchObservedRunningTime="2026-07-25 01:12:04.693997077 +0000 UTC m=+65.815482955" Jul 25 01:12:04.709580 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:04.709558 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9sn77_becef38c-52d9-4baa-bfe9-30f712afabdc/dns-node-resolver/0.log" Jul 25 01:12:05.112268 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.112230 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs\") pod \"network-metrics-daemon-xtmst\" (UID: \"55676d4a-8fb7-483b-b880-232cc2d10465\") " pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:12:05.114516 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.114500 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 25 01:12:05.124785 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.124756 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55676d4a-8fb7-483b-b880-232cc2d10465-metrics-certs\") pod \"network-metrics-daemon-xtmst\" (UID: \"55676d4a-8fb7-483b-b880-232cc2d10465\") " pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:12:05.213238 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.213212 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret\") pod \"global-pull-secret-syncer-jm7hh\" (UID: \"62f24f02-b14f-4633-a4e0-bed308f33ada\") " pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:12:05.213400 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.213265 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6464s\" (UniqueName: \"kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s\") pod \"network-check-target-2498t\" (UID: \"6727d4bf-1e34-43aa-8766-cc4e224468b3\") " pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:12:05.215286 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.215262 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 25 01:12:05.215402 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.215352 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 25 01:12:05.225538 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.225508 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 25 01:12:05.226217 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.226198 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f24f02-b14f-4633-a4e0-bed308f33ada-original-pull-secret\") pod \"global-pull-secret-syncer-jm7hh\" (UID: \"62f24f02-b14f-4633-a4e0-bed308f33ada\") " pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:12:05.237116 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.237097 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6464s\" (UniqueName: \"kubernetes.io/projected/6727d4bf-1e34-43aa-8766-cc4e224468b3-kube-api-access-6464s\") pod \"network-check-target-2498t\" (UID: \"6727d4bf-1e34-43aa-8766-cc4e224468b3\") " pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:12:05.290934 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.290915 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f4ttp\"" Jul 25 01:12:05.299531 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.299516 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:12:05.309743 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.309726 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d99ff7995-kcnlv_0d77b32a-87c3-4ea9-9d10-5eb70418a914/registry/0.log" Jul 25 01:12:05.388196 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.388166 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm7hh" Jul 25 01:12:05.395700 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.395673 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-djkst\"" Jul 25 01:12:05.403961 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.403937 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtmst" Jul 25 01:12:05.426917 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.426894 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2498t"] Jul 25 01:12:05.428447 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:12:05.428422 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6727d4bf_1e34_43aa_8766_cc4e224468b3.slice/crio-2a4992ad854fe3860df8a33bfe5358e82ccfaaf4487105def5b173cce87f4ba0 WatchSource:0}: Error finding container 2a4992ad854fe3860df8a33bfe5358e82ccfaaf4487105def5b173cce87f4ba0: Status 404 returned error can't find the container with id 2a4992ad854fe3860df8a33bfe5358e82ccfaaf4487105def5b173cce87f4ba0 Jul 25 01:12:05.510570 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.510547 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-69dbd56bdd-clccr_c3a5ae83-b4b8-47c6-8757-d1212f5ad638/registry/0.log" Jul 25 01:12:05.528214 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.528177 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jm7hh"] Jul 25 01:12:05.531484 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:12:05.531453 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f24f02_b14f_4633_a4e0_bed308f33ada.slice/crio-fb95ba1d5b91a97853d27df06342f74b882955f078160ce8d15be76ca07f534e WatchSource:0}: Error finding container fb95ba1d5b91a97853d27df06342f74b882955f078160ce8d15be76ca07f534e: Status 404 returned error can't find the container with id fb95ba1d5b91a97853d27df06342f74b882955f078160ce8d15be76ca07f534e Jul 25 01:12:05.546849 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.546826 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xtmst"] Jul 25 01:12:05.549648 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:12:05.549622 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55676d4a_8fb7_483b_b880_232cc2d10465.slice/crio-dd50630cda8f7aa8b1d074d6de5ac7b0c5a447bc1fd34392e9da43aeb0c28a9c WatchSource:0}: Error finding container dd50630cda8f7aa8b1d074d6de5ac7b0c5a447bc1fd34392e9da43aeb0c28a9c: Status 404 returned error can't find the container with id dd50630cda8f7aa8b1d074d6de5ac7b0c5a447bc1fd34392e9da43aeb0c28a9c Jul 25 01:12:05.689218 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.689180 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xtmst" event={"ID":"55676d4a-8fb7-483b-b880-232cc2d10465","Type":"ContainerStarted","Data":"dd50630cda8f7aa8b1d074d6de5ac7b0c5a447bc1fd34392e9da43aeb0c28a9c"} Jul 25 01:12:05.690330 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.690302 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jm7hh" event={"ID":"62f24f02-b14f-4633-a4e0-bed308f33ada","Type":"ContainerStarted","Data":"fb95ba1d5b91a97853d27df06342f74b882955f078160ce8d15be76ca07f534e"} Jul 25 01:12:05.691444 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:05.691421 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2498t" event={"ID":"6727d4bf-1e34-43aa-8766-cc4e224468b3","Type":"ContainerStarted","Data":"2a4992ad854fe3860df8a33bfe5358e82ccfaaf4487105def5b173cce87f4ba0"} Jul 25 01:12:06.309914 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:06.309880 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tb5g6_dfd8d67b-61d5-4a13-b319-3a64776c0a5b/node-ca/0.log" Jul 25 01:12:06.511267 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:06.511237 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cdd8d58d5-6v7sn_207f125a-e453-4137-95d2-29c48898e3bf/router/0.log" Jul 25 01:12:06.625404 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:06.625319 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:06.625593 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:06.625426 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:06.625593 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:12:06.625549 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 25 01:12:06.625721 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:12:06.625602 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-insights-runtime-extractor-tls podName:25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b nodeName:}" failed. No retries permitted until 2026-07-25 01:12:10.62558711 +0000 UTC m=+71.747072938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnkn9" (UID: "25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b") : secret "insights-runtime-extractor-tls" not found Jul 25 01:12:06.625721 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:12:06.625649 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 25 01:12:06.625721 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:12:06.625680 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-kube-rbac-proxy-cm podName:25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b nodeName:}" failed. No retries permitted until 2026-07-25 01:12:10.625670733 +0000 UTC m=+71.747156561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnkn9" (UID: "25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b") : configmap "kube-rbac-proxy" not found Jul 25 01:12:07.109671 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:07.109639 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zc4st_a8916f19-f4e7-4bbd-a0cb-49150afe5542/serve-healthcheck-canary/0.log" Jul 25 01:12:07.300703 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:07.300671 2554 patch_prober.go:28] interesting pod/image-registry-69dbd56bdd-clccr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 25 01:12:07.300883 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:07.300730 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" podUID="c3a5ae83-b4b8-47c6-8757-d1212f5ad638" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 25 01:12:07.702419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:07.702366 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xtmst" event={"ID":"55676d4a-8fb7-483b-b880-232cc2d10465","Type":"ContainerStarted","Data":"825d780428d13c8619c64cd34cffdd471f9f13d7aad0f6e48d82696a01b8443c"} Jul 25 01:12:07.702889 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:07.702437 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xtmst" event={"ID":"55676d4a-8fb7-483b-b880-232cc2d10465","Type":"ContainerStarted","Data":"937a1f5e569ba99d4aceff36185951d81dbc20bc9874cd22ed51a7032a7256eb"} Jul 25 01:12:07.721703 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:07.721643 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xtmst" podStartSLOduration=67.093461244 podStartE2EDuration="1m8.721623331s" totalImagesPullingTime="1.628162087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:07.719987072 +0000 UTC m=+68.841472923" watchObservedRunningTime="2026-07-25 01:12:07.721623331 +0000 UTC m=+68.843109182" Jul 25 01:12:10.664912 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:10.664875 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:10.665298 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:10.664922 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:10.666250 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:10.665864 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:10.670688 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:10.670668 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnkn9\" (UID: \"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b\") " pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:10.716772 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:10.716751 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gnkn9" Jul 25 01:12:10.891395 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:10.891363 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gnkn9"] Jul 25 01:12:10.896918 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:12:10.896882 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25beb3ea_d0d4_4d1b_bc89_b202ec6b1a7b.slice/crio-4407bf23b1829528c4c1fcdd1beb4d7454d4a8ad960c0bf6dc4e890b6d5eb8e2 WatchSource:0}: Error finding container 4407bf23b1829528c4c1fcdd1beb4d7454d4a8ad960c0bf6dc4e890b6d5eb8e2: Status 404 returned error can't find the container with id 4407bf23b1829528c4c1fcdd1beb4d7454d4a8ad960c0bf6dc4e890b6d5eb8e2 Jul 25 01:12:11.719994 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:11.719429 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jm7hh" event={"ID":"62f24f02-b14f-4633-a4e0-bed308f33ada","Type":"ContainerStarted","Data":"dcc760c34ab47bcb2d51f5ddce90e2177bbd527ee3d3ee3b814efdddbe57e0d7"} Jul 25 01:12:11.722065 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:11.722034 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2498t" event={"ID":"6727d4bf-1e34-43aa-8766-cc4e224468b3","Type":"ContainerStarted","Data":"c1be967ede2c32e8fb5ac24fb7aa8368e368b7f4b0b52f08f4c3285c823985cb"} Jul 25 01:12:11.722204 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:11.722087 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:12:11.723996 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:11.723962 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnkn9" event={"ID":"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b","Type":"ContainerStarted","Data":"4407bf23b1829528c4c1fcdd1beb4d7454d4a8ad960c0bf6dc4e890b6d5eb8e2"} Jul 25 01:12:11.750929 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:11.750874 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jm7hh" podStartSLOduration=67.58313948 podStartE2EDuration="1m12.750853543s" totalImagesPullingTime="5.167714063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:11.750393161 +0000 UTC m=+72.871879012" watchObservedRunningTime="2026-07-25 01:12:11.750853543 +0000 UTC m=+72.872339394" Jul 25 01:12:11.780034 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:11.779985 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2498t" podStartSLOduration=67.521795197 podStartE2EDuration="1m12.779966681s" totalImagesPullingTime="5.258171484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:11.779734497 +0000 UTC m=+72.901220346" watchObservedRunningTime="2026-07-25 01:12:11.779966681 +0000 UTC m=+72.901452535" Jul 25 01:12:13.122729 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.122692 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-z7jn5"] Jul 25 01:12:13.126584 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.126102 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.130270 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.130247 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 25 01:12:13.130678 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.130648 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tqbdc\"" Jul 25 01:12:13.130795 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.130720 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 25 01:12:13.131063 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.131035 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 25 01:12:13.131229 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.131217 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 25 01:12:13.188055 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.188024 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-node-exporter-tls\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.188317 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.188061 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzg8h\" (UniqueName: \"kubernetes.io/projected/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-kube-api-access-kzg8h\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.188317 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.188081 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-sys\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.188317 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.188096 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-root\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.188317 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.188168 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-node-exporter-textfile\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.188317 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.188188 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-node-exporter-accelerators-collector-config\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.188317 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.188222 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-node-exporter-wtmp\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.188641 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.188340 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-metrics-client-ca\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.188641 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.188381 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.289371 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.289329 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-node-exporter-wtmp\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.289546 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.289384 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-metrics-client-ca\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.289546 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.289424 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.289546 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.289483 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-node-exporter-tls\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.289546 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.289523 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-node-exporter-wtmp\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.289753 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.289603 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzg8h\" (UniqueName: \"kubernetes.io/projected/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-kube-api-access-kzg8h\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.289753 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.289643 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-sys\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.289753 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.289668 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-root\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.289928 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.289711 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-node-exporter-textfile\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.290262 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.289788 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-node-exporter-accelerators-collector-config\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.290368 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.289888 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-root\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.290368 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.289922 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-sys\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.290368 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.290177 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-node-exporter-textfile\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.290368 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.290203 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-metrics-client-ca\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.290368 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.290359 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-node-exporter-accelerators-collector-config\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.292838 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.292784 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-node-exporter-tls\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.292929 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.292874 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.298602 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.298584 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzg8h\" (UniqueName: \"kubernetes.io/projected/5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7-kube-api-access-kzg8h\") pod \"node-exporter-z7jn5\" (UID: \"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7\") " pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:13.438773 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:13.438740 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z7jn5" Jul 25 01:12:15.957582 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:12:15.957550 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad0b383_9ccc_41bf_aa1e_809f49f6b7c7.slice/crio-da0441006c00f4fae3e55fff3ec49a3e6239914c5f51286be96ff21c3382405d WatchSource:0}: Error finding container da0441006c00f4fae3e55fff3ec49a3e6239914c5f51286be96ff21c3382405d: Status 404 returned error can't find the container with id da0441006c00f4fae3e55fff3ec49a3e6239914c5f51286be96ff21c3382405d Jul 25 01:12:16.749274 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:16.749180 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnkn9" event={"ID":"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b","Type":"ContainerStarted","Data":"766d2da7a90173f0de1ee6eb209394f4942ff5568fcec9ce3888d206da3f65a7"} Jul 25 01:12:16.750790 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:16.750722 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z7jn5" event={"ID":"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7","Type":"ContainerStarted","Data":"da0441006c00f4fae3e55fff3ec49a3e6239914c5f51286be96ff21c3382405d"} Jul 25 01:12:17.298085 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:17.298058 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:12:17.757384 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:17.757350 2554 generic.go:356] "Generic (PLEG): container finished" podID="5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7" containerID="7abbcf2cd3e856e4d1af3c3248cd5d94ac42a4f201dbc696d4ea207a7f247b41" exitCode=0 Jul 25 01:12:17.757569 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:17.757401 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z7jn5" event={"ID":"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7","Type":"ContainerDied","Data":"7abbcf2cd3e856e4d1af3c3248cd5d94ac42a4f201dbc696d4ea207a7f247b41"} Jul 25 01:12:19.765420 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:19.765281 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnkn9" event={"ID":"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b","Type":"ContainerStarted","Data":"a94ede489bdcf7cd3a0e1ebbd4ee9f0fc0806b6e1dfd17e3b31c062ba231d2b6"} Jul 25 01:12:19.767512 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:19.767487 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z7jn5" event={"ID":"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7","Type":"ContainerStarted","Data":"f3a1f74d555cd3ac297f9fd733a368ca950620c0ecfd6bd0491f74c99a691a71"} Jul 25 01:12:19.767644 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:19.767531 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z7jn5" event={"ID":"5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7","Type":"ContainerStarted","Data":"02efd7f62be8723d65cfd9612f2d3838a02a2db5babe9bc648bf77afdbd88f00"} Jul 25 01:12:19.787939 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:19.787898 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-z7jn5" podStartSLOduration=5.58037183 podStartE2EDuration="6.787883258s" totalImagesPullingTime="1.126388224s" totalInitContainerRuntime="81.123204ms" isStatefulPod=true podCreationTimestamp="2026-07-25 01:12:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:19.786192487 +0000 UTC m=+80.907678338" watchObservedRunningTime="2026-07-25 01:12:19.787883258 +0000 UTC m=+80.909369107" Jul 25 01:12:20.662554 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:20.662522 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d99ff7995-kcnlv" Jul 25 01:12:22.311850 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:22.311765 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" podUID="c3a5ae83-b4b8-47c6-8757-d1212f5ad638" containerName="registry" containerID="cri-o://b1f39897a02bf2eaa83358d47cc3cfe0fcd44389e362006c2cabe8f3e13ebe5e" gracePeriod=30 Jul 25 01:12:23.134002 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.133980 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:12:23.278445 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.278350 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-trusted-ca\") pod \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " Jul 25 01:12:23.278445 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.278422 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-ca-trust-extracted\") pod \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " Jul 25 01:12:23.278688 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.278490 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6wwr\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-kube-api-access-q6wwr\") pod \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " Jul 25 01:12:23.278688 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.278548 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-image-registry-private-configuration\") pod \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " Jul 25 01:12:23.278688 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.278579 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls\") pod \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " Jul 25 01:12:23.278688 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.278607 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-bound-sa-token\") pod \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " Jul 25 01:12:23.278688 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.278639 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-certificates\") pod \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " Jul 25 01:12:23.278688 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.278674 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-installation-pull-secrets\") pod \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\" (UID: \"c3a5ae83-b4b8-47c6-8757-d1212f5ad638\") " Jul 25 01:12:23.278988 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.278834 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c3a5ae83-b4b8-47c6-8757-d1212f5ad638" (UID: "c3a5ae83-b4b8-47c6-8757-d1212f5ad638"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 01:12:23.278988 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.278959 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-trusted-ca\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:12:23.279346 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.279306 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c3a5ae83-b4b8-47c6-8757-d1212f5ad638" (UID: "c3a5ae83-b4b8-47c6-8757-d1212f5ad638"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 01:12:23.281739 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.281677 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c3a5ae83-b4b8-47c6-8757-d1212f5ad638" (UID: "c3a5ae83-b4b8-47c6-8757-d1212f5ad638"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:12:23.281739 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.281694 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c3a5ae83-b4b8-47c6-8757-d1212f5ad638" (UID: "c3a5ae83-b4b8-47c6-8757-d1212f5ad638"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 01:12:23.281936 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.281761 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c3a5ae83-b4b8-47c6-8757-d1212f5ad638" (UID: "c3a5ae83-b4b8-47c6-8757-d1212f5ad638"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:12:23.281936 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.281774 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c3a5ae83-b4b8-47c6-8757-d1212f5ad638" (UID: "c3a5ae83-b4b8-47c6-8757-d1212f5ad638"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 01:12:23.281936 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.281848 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-kube-api-access-q6wwr" (OuterVolumeSpecName: "kube-api-access-q6wwr") pod "c3a5ae83-b4b8-47c6-8757-d1212f5ad638" (UID: "c3a5ae83-b4b8-47c6-8757-d1212f5ad638"). InnerVolumeSpecName "kube-api-access-q6wwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:12:23.287194 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.287173 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c3a5ae83-b4b8-47c6-8757-d1212f5ad638" (UID: "c3a5ae83-b4b8-47c6-8757-d1212f5ad638"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:12:23.379841 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.379593 2554 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-image-registry-private-configuration\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:12:23.379841 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.379621 2554 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:12:23.379841 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.379635 2554 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-bound-sa-token\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:12:23.379841 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.379646 2554 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-registry-certificates\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:12:23.379841 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.379659 2554 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-installation-pull-secrets\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:12:23.379841 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.379673 2554 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-ca-trust-extracted\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:12:23.379841 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.379685 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6wwr\" (UniqueName: \"kubernetes.io/projected/c3a5ae83-b4b8-47c6-8757-d1212f5ad638-kube-api-access-q6wwr\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:12:23.785077 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.785041 2554 generic.go:356] "Generic (PLEG): container finished" podID="c3a5ae83-b4b8-47c6-8757-d1212f5ad638" containerID="b1f39897a02bf2eaa83358d47cc3cfe0fcd44389e362006c2cabe8f3e13ebe5e" exitCode=0 Jul 25 01:12:23.785230 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.785106 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" Jul 25 01:12:23.785230 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.785131 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" event={"ID":"c3a5ae83-b4b8-47c6-8757-d1212f5ad638","Type":"ContainerDied","Data":"b1f39897a02bf2eaa83358d47cc3cfe0fcd44389e362006c2cabe8f3e13ebe5e"} Jul 25 01:12:23.785230 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.785190 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69dbd56bdd-clccr" event={"ID":"c3a5ae83-b4b8-47c6-8757-d1212f5ad638","Type":"ContainerDied","Data":"c3aef7fc4e5e78cf09ab9d6f9113a78fb509078c27ba51c303dc3740eb861187"} Jul 25 01:12:23.785230 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.785217 2554 scope.go:117] "RemoveContainer" containerID="b1f39897a02bf2eaa83358d47cc3cfe0fcd44389e362006c2cabe8f3e13ebe5e" Jul 25 01:12:23.787730 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.787706 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnkn9" event={"ID":"25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b","Type":"ContainerStarted","Data":"0a32870420287142108eeb04c2663662a26356285831e6248a220f544b8688d6"} Jul 25 01:12:23.793731 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.793710 2554 scope.go:117] "RemoveContainer" containerID="b1f39897a02bf2eaa83358d47cc3cfe0fcd44389e362006c2cabe8f3e13ebe5e" Jul 25 01:12:23.794027 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:12:23.794005 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f39897a02bf2eaa83358d47cc3cfe0fcd44389e362006c2cabe8f3e13ebe5e\": container with ID starting with b1f39897a02bf2eaa83358d47cc3cfe0fcd44389e362006c2cabe8f3e13ebe5e not found: ID does not exist" containerID="b1f39897a02bf2eaa83358d47cc3cfe0fcd44389e362006c2cabe8f3e13ebe5e" Jul 25 01:12:23.794100 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.794034 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f39897a02bf2eaa83358d47cc3cfe0fcd44389e362006c2cabe8f3e13ebe5e"} err="failed to get container status \"b1f39897a02bf2eaa83358d47cc3cfe0fcd44389e362006c2cabe8f3e13ebe5e\": rpc error: code = NotFound desc = could not find container \"b1f39897a02bf2eaa83358d47cc3cfe0fcd44389e362006c2cabe8f3e13ebe5e\": container with ID starting with b1f39897a02bf2eaa83358d47cc3cfe0fcd44389e362006c2cabe8f3e13ebe5e not found: ID does not exist" Jul 25 01:12:23.808103 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.807741 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gnkn9" podStartSLOduration=9.806135539 podStartE2EDuration="21.807724171s" totalImagesPullingTime="12.001588632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:12:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:23.806928074 +0000 UTC m=+84.928413918" watchObservedRunningTime="2026-07-25 01:12:23.807724171 +0000 UTC m=+84.929210022" Jul 25 01:12:23.821169 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.821147 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69dbd56bdd-clccr"] Jul 25 01:12:23.824632 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:23.824613 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69dbd56bdd-clccr"] Jul 25 01:12:25.379285 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:25.379254 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3a5ae83-b4b8-47c6-8757-d1212f5ad638" path="/var/lib/kubelet/pods/c3a5ae83-b4b8-47c6-8757-d1212f5ad638/volumes" Jul 25 01:12:42.730913 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:12:42.730882 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2498t" Jul 25 01:15:28.601682 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:28.601596 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc"] Jul 25 01:15:28.602153 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:28.601969 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3a5ae83-b4b8-47c6-8757-d1212f5ad638" containerName="registry" Jul 25 01:15:28.602153 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:28.601987 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3a5ae83-b4b8-47c6-8757-d1212f5ad638" containerName="registry" Jul 25 01:15:28.602153 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:28.602033 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3a5ae83-b4b8-47c6-8757-d1212f5ad638" containerName="registry" Jul 25 01:15:28.604988 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:28.604969 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc" Jul 25 01:15:28.606987 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:28.606966 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 25 01:15:28.607102 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:28.607003 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 25 01:15:28.607580 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:28.607566 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wthp2\"" Jul 25 01:15:28.614574 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:28.614554 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc"] Jul 25 01:15:28.671680 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:28.671655 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aaa0fa4a-d212-4cf4-a859-c7a95bc98abe-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc\" (UID: \"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc" Jul 25 01:15:28.671776 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:28.671683 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7gnr\" (UniqueName: \"kubernetes.io/projected/aaa0fa4a-d212-4cf4-a859-c7a95bc98abe-kube-api-access-r7gnr\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc\" (UID: \"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc" Jul 25 01:15:28.671776 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:28.671708 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaa0fa4a-d212-4cf4-a859-c7a95bc98abe-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc\" (UID: \"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc" Jul 25 01:15:28.772170 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:28.772138 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aaa0fa4a-d212-4cf4-a859-c7a95bc98abe-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc\" (UID: \"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc" Jul 25 01:15:28.772170 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:28.772174 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7gnr\" (UniqueName: \"kubernetes.io/projected/aaa0fa4a-d212-4cf4-a859-c7a95bc98abe-kube-api-access-r7gnr\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc\" (UID: \"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc" Jul 25 01:15:28.772308 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:28.772195 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaa0fa4a-d212-4cf4-a859-c7a95bc98abe-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc\" (UID: \"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc" Jul 25 01:15:28.772484 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:28.772457 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aaa0fa4a-d212-4cf4-a859-c7a95bc98abe-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc\" (UID: \"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc" Jul 25 01:15:28.772536 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:28.772521 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaa0fa4a-d212-4cf4-a859-c7a95bc98abe-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc\" (UID: \"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc" Jul 25 01:15:28.779646 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:28.779619 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7gnr\" (UniqueName: \"kubernetes.io/projected/aaa0fa4a-d212-4cf4-a859-c7a95bc98abe-kube-api-access-r7gnr\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc\" (UID: \"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc" Jul 25 01:15:28.914529 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:28.914508 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc" Jul 25 01:15:29.043686 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:29.043614 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc"] Jul 25 01:15:29.410875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:29.410785 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc" event={"ID":"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe","Type":"ContainerStarted","Data":"c584f7bf7b29e0e7afed4abaa360aaf1b3cb59cae99218af5e07d41224b8af7e"} Jul 25 01:15:35.432928 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:35.432884 2554 generic.go:356] "Generic (PLEG): container finished" podID="aaa0fa4a-d212-4cf4-a859-c7a95bc98abe" containerID="499b2665eee6399088d03187f854bf0dea6c5e33e9f74aa4a9558bf2f67301a5" exitCode=0 Jul 25 01:15:35.433331 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:35.432977 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc" event={"ID":"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe","Type":"ContainerDied","Data":"499b2665eee6399088d03187f854bf0dea6c5e33e9f74aa4a9558bf2f67301a5"} Jul 25 01:15:38.444634 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:38.444565 2554 generic.go:356] "Generic (PLEG): container finished" podID="aaa0fa4a-d212-4cf4-a859-c7a95bc98abe" containerID="a083b8011ac3629f022ca732cb424d3b24957194ccead2433fd6799503f1e29f" exitCode=0 Jul 25 01:15:38.444634 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:38.444611 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc" event={"ID":"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe","Type":"ContainerDied","Data":"a083b8011ac3629f022ca732cb424d3b24957194ccead2433fd6799503f1e29f"} Jul 25 01:15:45.471789 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:45.471753 2554 generic.go:356] "Generic (PLEG): container finished" podID="aaa0fa4a-d212-4cf4-a859-c7a95bc98abe" containerID="e8e4d1c54349e605bc340b632d70d8e64bf29198d9aef19e6530b82dc75b8871" exitCode=0 Jul 25 01:15:45.472182 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:45.471815 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc" event={"ID":"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe","Type":"ContainerDied","Data":"e8e4d1c54349e605bc340b632d70d8e64bf29198d9aef19e6530b82dc75b8871"} Jul 25 01:15:46.601695 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:46.601670 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc" Jul 25 01:15:46.714728 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:46.714698 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaa0fa4a-d212-4cf4-a859-c7a95bc98abe-bundle\") pod \"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe\" (UID: \"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe\") " Jul 25 01:15:46.714886 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:46.714753 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aaa0fa4a-d212-4cf4-a859-c7a95bc98abe-util\") pod \"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe\" (UID: \"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe\") " Jul 25 01:15:46.714886 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:46.714774 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7gnr\" (UniqueName: \"kubernetes.io/projected/aaa0fa4a-d212-4cf4-a859-c7a95bc98abe-kube-api-access-r7gnr\") pod \"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe\" (UID: \"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe\") " Jul 25 01:15:46.715516 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:46.715490 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa0fa4a-d212-4cf4-a859-c7a95bc98abe-bundle" (OuterVolumeSpecName: "bundle") pod "aaa0fa4a-d212-4cf4-a859-c7a95bc98abe" (UID: "aaa0fa4a-d212-4cf4-a859-c7a95bc98abe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:15:46.717037 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:46.717016 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa0fa4a-d212-4cf4-a859-c7a95bc98abe-kube-api-access-r7gnr" (OuterVolumeSpecName: "kube-api-access-r7gnr") pod "aaa0fa4a-d212-4cf4-a859-c7a95bc98abe" (UID: "aaa0fa4a-d212-4cf4-a859-c7a95bc98abe"). InnerVolumeSpecName "kube-api-access-r7gnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:15:46.721197 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:46.721173 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa0fa4a-d212-4cf4-a859-c7a95bc98abe-util" (OuterVolumeSpecName: "util") pod "aaa0fa4a-d212-4cf4-a859-c7a95bc98abe" (UID: "aaa0fa4a-d212-4cf4-a859-c7a95bc98abe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:15:46.815232 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:46.815171 2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaa0fa4a-d212-4cf4-a859-c7a95bc98abe-bundle\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:15:46.815232 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:46.815191 2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aaa0fa4a-d212-4cf4-a859-c7a95bc98abe-util\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:15:46.815232 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:46.815199 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7gnr\" (UniqueName: \"kubernetes.io/projected/aaa0fa4a-d212-4cf4-a859-c7a95bc98abe-kube-api-access-r7gnr\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:15:47.480367 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:47.480337 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc" event={"ID":"aaa0fa4a-d212-4cf4-a859-c7a95bc98abe","Type":"ContainerDied","Data":"c584f7bf7b29e0e7afed4abaa360aaf1b3cb59cae99218af5e07d41224b8af7e"} Jul 25 01:15:47.480532 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:47.480386 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f9msc" Jul 25 01:15:47.480532 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:47.480389 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c584f7bf7b29e0e7afed4abaa360aaf1b3cb59cae99218af5e07d41224b8af7e" Jul 25 01:15:59.295180 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:15:59.295152 2554 kubelet.go:1648] "Image garbage collection succeeded" Jul 25 01:16:04.995077 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:04.995043 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq"] Jul 25 01:16:04.997129 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:04.995383 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa0fa4a-d212-4cf4-a859-c7a95bc98abe" containerName="util" Jul 25 01:16:04.997129 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:04.995400 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa0fa4a-d212-4cf4-a859-c7a95bc98abe" containerName="util" Jul 25 01:16:04.997129 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:04.995419 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa0fa4a-d212-4cf4-a859-c7a95bc98abe" containerName="pull" Jul 25 01:16:04.997129 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:04.995426 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa0fa4a-d212-4cf4-a859-c7a95bc98abe" containerName="pull" Jul 25 01:16:04.997129 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:04.995445 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa0fa4a-d212-4cf4-a859-c7a95bc98abe" containerName="extract" Jul 25 01:16:04.997129 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:04.995453 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa0fa4a-d212-4cf4-a859-c7a95bc98abe" containerName="extract" Jul 25 01:16:04.997129 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:04.995508 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa0fa4a-d212-4cf4-a859-c7a95bc98abe" containerName="extract" Jul 25 01:16:04.997876 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:04.997858 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq" Jul 25 01:16:04.999988 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:04.999958 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 25 01:16:05.000102 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:04.999989 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 25 01:16:05.000102 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:05.000072 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wthp2\"" Jul 25 01:16:05.004969 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:05.004947 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq"] Jul 25 01:16:05.139288 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:05.139259 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6931c907-8cda-4eec-b650-6c910452b0c6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq\" (UID: \"6931c907-8cda-4eec-b650-6c910452b0c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq" Jul 25 01:16:05.139288 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:05.139291 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6931c907-8cda-4eec-b650-6c910452b0c6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq\" (UID: \"6931c907-8cda-4eec-b650-6c910452b0c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq" Jul 25 01:16:05.139470 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:05.139311 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4trm\" (UniqueName: \"kubernetes.io/projected/6931c907-8cda-4eec-b650-6c910452b0c6-kube-api-access-z4trm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq\" (UID: \"6931c907-8cda-4eec-b650-6c910452b0c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq" Jul 25 01:16:05.240627 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:05.240599 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6931c907-8cda-4eec-b650-6c910452b0c6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq\" (UID: \"6931c907-8cda-4eec-b650-6c910452b0c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq" Jul 25 01:16:05.240760 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:05.240645 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4trm\" (UniqueName: \"kubernetes.io/projected/6931c907-8cda-4eec-b650-6c910452b0c6-kube-api-access-z4trm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq\" (UID: \"6931c907-8cda-4eec-b650-6c910452b0c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq" Jul 25 01:16:05.240760 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:05.240715 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6931c907-8cda-4eec-b650-6c910452b0c6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq\" (UID: \"6931c907-8cda-4eec-b650-6c910452b0c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq" Jul 25 01:16:05.241003 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:05.240984 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6931c907-8cda-4eec-b650-6c910452b0c6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq\" (UID: \"6931c907-8cda-4eec-b650-6c910452b0c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq" Jul 25 01:16:05.241067 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:05.241048 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6931c907-8cda-4eec-b650-6c910452b0c6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq\" (UID: \"6931c907-8cda-4eec-b650-6c910452b0c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq" Jul 25 01:16:05.249157 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:05.249098 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4trm\" (UniqueName: \"kubernetes.io/projected/6931c907-8cda-4eec-b650-6c910452b0c6-kube-api-access-z4trm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq\" (UID: \"6931c907-8cda-4eec-b650-6c910452b0c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq" Jul 25 01:16:05.307304 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:05.307283 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq" Jul 25 01:16:05.455778 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:05.455630 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq"] Jul 25 01:16:05.458303 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:16:05.458268 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6931c907_8cda_4eec_b650_6c910452b0c6.slice/crio-a97741abd54b9cf235c2e871ec04fe3cf4c05108e2d266f4efcb28e7080c0468 WatchSource:0}: Error finding container a97741abd54b9cf235c2e871ec04fe3cf4c05108e2d266f4efcb28e7080c0468: Status 404 returned error can't find the container with id a97741abd54b9cf235c2e871ec04fe3cf4c05108e2d266f4efcb28e7080c0468 Jul 25 01:16:05.542455 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:05.542421 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq" event={"ID":"6931c907-8cda-4eec-b650-6c910452b0c6","Type":"ContainerStarted","Data":"1b8548658bcd7e10eec062a56bbdafebd19017637ecb63d785e9b89dbcbf06c5"} Jul 25 01:16:05.542614 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:05.542468 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq" event={"ID":"6931c907-8cda-4eec-b650-6c910452b0c6","Type":"ContainerStarted","Data":"a97741abd54b9cf235c2e871ec04fe3cf4c05108e2d266f4efcb28e7080c0468"} Jul 25 01:16:06.548434 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:06.548347 2554 generic.go:356] "Generic (PLEG): container finished" podID="6931c907-8cda-4eec-b650-6c910452b0c6" containerID="1b8548658bcd7e10eec062a56bbdafebd19017637ecb63d785e9b89dbcbf06c5" exitCode=0 Jul 25 01:16:06.548434 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:06.548412 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq" event={"ID":"6931c907-8cda-4eec-b650-6c910452b0c6","Type":"ContainerDied","Data":"1b8548658bcd7e10eec062a56bbdafebd19017637ecb63d785e9b89dbcbf06c5"} Jul 25 01:16:06.549412 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:06.549397 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 25 01:16:09.561195 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:09.561163 2554 generic.go:356] "Generic (PLEG): container finished" podID="6931c907-8cda-4eec-b650-6c910452b0c6" containerID="d9dbd2dcf2415a26e80705009d0fc0d9a5d2fb66e61bdb021b3b4f274eb60ed3" exitCode=0 Jul 25 01:16:09.561576 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:09.561200 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq" event={"ID":"6931c907-8cda-4eec-b650-6c910452b0c6","Type":"ContainerDied","Data":"d9dbd2dcf2415a26e80705009d0fc0d9a5d2fb66e61bdb021b3b4f274eb60ed3"} Jul 25 01:16:10.566922 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:10.566886 2554 generic.go:356] "Generic (PLEG): container finished" podID="6931c907-8cda-4eec-b650-6c910452b0c6" containerID="b948fca83049014fac7c422697e068340077f6a31b373cffb7382e47f828b926" exitCode=0 Jul 25 01:16:10.567296 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:10.566943 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq" event={"ID":"6931c907-8cda-4eec-b650-6c910452b0c6","Type":"ContainerDied","Data":"b948fca83049014fac7c422697e068340077f6a31b373cffb7382e47f828b926"} Jul 25 01:16:11.692776 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:11.692753 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq" Jul 25 01:16:11.788659 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:11.788633 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6931c907-8cda-4eec-b650-6c910452b0c6-util\") pod \"6931c907-8cda-4eec-b650-6c910452b0c6\" (UID: \"6931c907-8cda-4eec-b650-6c910452b0c6\") " Jul 25 01:16:11.788776 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:11.788701 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4trm\" (UniqueName: \"kubernetes.io/projected/6931c907-8cda-4eec-b650-6c910452b0c6-kube-api-access-z4trm\") pod \"6931c907-8cda-4eec-b650-6c910452b0c6\" (UID: \"6931c907-8cda-4eec-b650-6c910452b0c6\") " Jul 25 01:16:11.788776 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:11.788723 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6931c907-8cda-4eec-b650-6c910452b0c6-bundle\") pod \"6931c907-8cda-4eec-b650-6c910452b0c6\" (UID: \"6931c907-8cda-4eec-b650-6c910452b0c6\") " Jul 25 01:16:11.789130 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:11.789112 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6931c907-8cda-4eec-b650-6c910452b0c6-bundle" (OuterVolumeSpecName: "bundle") pod "6931c907-8cda-4eec-b650-6c910452b0c6" (UID: "6931c907-8cda-4eec-b650-6c910452b0c6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:16:11.790956 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:11.790929 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6931c907-8cda-4eec-b650-6c910452b0c6-kube-api-access-z4trm" (OuterVolumeSpecName: "kube-api-access-z4trm") pod "6931c907-8cda-4eec-b650-6c910452b0c6" (UID: "6931c907-8cda-4eec-b650-6c910452b0c6"). InnerVolumeSpecName "kube-api-access-z4trm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:16:11.794205 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:11.794155 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6931c907-8cda-4eec-b650-6c910452b0c6-util" (OuterVolumeSpecName: "util") pod "6931c907-8cda-4eec-b650-6c910452b0c6" (UID: "6931c907-8cda-4eec-b650-6c910452b0c6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:16:11.889854 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:11.889779 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4trm\" (UniqueName: \"kubernetes.io/projected/6931c907-8cda-4eec-b650-6c910452b0c6-kube-api-access-z4trm\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:16:11.889854 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:11.889821 2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6931c907-8cda-4eec-b650-6c910452b0c6-bundle\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:16:11.889854 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:11.889832 2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6931c907-8cda-4eec-b650-6c910452b0c6-util\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:16:12.576578 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:12.576549 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq" event={"ID":"6931c907-8cda-4eec-b650-6c910452b0c6","Type":"ContainerDied","Data":"a97741abd54b9cf235c2e871ec04fe3cf4c05108e2d266f4efcb28e7080c0468"} Jul 25 01:16:12.576738 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:12.576593 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a97741abd54b9cf235c2e871ec04fe3cf4c05108e2d266f4efcb28e7080c0468" Jul 25 01:16:12.576738 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:12.576595 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flg4gq" Jul 25 01:16:17.988632 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:17.988601 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4k854"] Jul 25 01:16:17.989022 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:17.988877 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6931c907-8cda-4eec-b650-6c910452b0c6" containerName="pull" Jul 25 01:16:17.989022 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:17.988889 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6931c907-8cda-4eec-b650-6c910452b0c6" containerName="pull" Jul 25 01:16:17.989022 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:17.988911 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6931c907-8cda-4eec-b650-6c910452b0c6" containerName="util" Jul 25 01:16:17.989022 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:17.988918 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6931c907-8cda-4eec-b650-6c910452b0c6" containerName="util" Jul 25 01:16:17.989022 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:17.988924 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6931c907-8cda-4eec-b650-6c910452b0c6" containerName="extract" Jul 25 01:16:17.989022 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:17.988929 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6931c907-8cda-4eec-b650-6c910452b0c6" containerName="extract" Jul 25 01:16:17.989022 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:17.988973 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6931c907-8cda-4eec-b650-6c910452b0c6" containerName="extract" Jul 25 01:16:17.990913 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:17.990897 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4k854" Jul 25 01:16:17.993120 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:17.993097 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 25 01:16:17.993120 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:17.993115 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 25 01:16:17.993629 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:17.993613 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6b726\"" Jul 25 01:16:17.999403 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:17.999381 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4k854"] Jul 25 01:16:18.140151 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:18.140124 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmvn4\" (UniqueName: \"kubernetes.io/projected/f01d3044-15ea-4857-a34d-f876f9bf0902-kube-api-access-xmvn4\") pod \"openshift-lws-operator-fd8ccff4c-4k854\" (UID: \"f01d3044-15ea-4857-a34d-f876f9bf0902\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4k854" Jul 25 01:16:18.140292 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:18.140187 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f01d3044-15ea-4857-a34d-f876f9bf0902-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4k854\" (UID: \"f01d3044-15ea-4857-a34d-f876f9bf0902\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4k854" Jul 25 01:16:18.241123 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:18.241052 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f01d3044-15ea-4857-a34d-f876f9bf0902-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4k854\" (UID: \"f01d3044-15ea-4857-a34d-f876f9bf0902\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4k854" Jul 25 01:16:18.241123 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:18.241095 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmvn4\" (UniqueName: \"kubernetes.io/projected/f01d3044-15ea-4857-a34d-f876f9bf0902-kube-api-access-xmvn4\") pod \"openshift-lws-operator-fd8ccff4c-4k854\" (UID: \"f01d3044-15ea-4857-a34d-f876f9bf0902\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4k854" Jul 25 01:16:18.241483 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:18.241465 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f01d3044-15ea-4857-a34d-f876f9bf0902-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4k854\" (UID: \"f01d3044-15ea-4857-a34d-f876f9bf0902\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4k854" Jul 25 01:16:18.250938 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:18.250916 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmvn4\" (UniqueName: \"kubernetes.io/projected/f01d3044-15ea-4857-a34d-f876f9bf0902-kube-api-access-xmvn4\") pod \"openshift-lws-operator-fd8ccff4c-4k854\" (UID: \"f01d3044-15ea-4857-a34d-f876f9bf0902\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4k854" Jul 25 01:16:18.299816 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:18.299778 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4k854" Jul 25 01:16:18.424008 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:18.423980 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4k854"] Jul 25 01:16:18.427393 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:16:18.427370 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf01d3044_15ea_4857_a34d_f876f9bf0902.slice/crio-3805783c4563e365456d88c4d7d58d6a26202d35a063142d9200ba92367c7d56 WatchSource:0}: Error finding container 3805783c4563e365456d88c4d7d58d6a26202d35a063142d9200ba92367c7d56: Status 404 returned error can't find the container with id 3805783c4563e365456d88c4d7d58d6a26202d35a063142d9200ba92367c7d56 Jul 25 01:16:18.598429 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:18.598350 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4k854" event={"ID":"f01d3044-15ea-4857-a34d-f876f9bf0902","Type":"ContainerStarted","Data":"3805783c4563e365456d88c4d7d58d6a26202d35a063142d9200ba92367c7d56"} Jul 25 01:16:20.607549 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:20.607508 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4k854" event={"ID":"f01d3044-15ea-4857-a34d-f876f9bf0902","Type":"ContainerStarted","Data":"c944c85ca9499b038f19d8cd39e970e9fbc0728303196e732582c609431232db"} Jul 25 01:16:20.623081 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:20.622719 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4k854" podStartSLOduration=1.5721637419999999 podStartE2EDuration="3.622700023s" totalImagesPullingTime="2.050536281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:16:20.621721227 +0000 UTC m=+321.743207078" watchObservedRunningTime="2026-07-25 01:16:20.622700023 +0000 UTC m=+321.744185874" Jul 25 01:16:23.286529 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:23.286495 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s"] Jul 25 01:16:23.288913 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:23.288898 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s" Jul 25 01:16:23.291003 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:23.290983 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 25 01:16:23.291603 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:23.291588 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wthp2\"" Jul 25 01:16:23.291674 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:23.291641 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 25 01:16:23.296856 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:23.296835 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s"] Jul 25 01:16:23.477815 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:23.477754 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/116470cb-2dae-4b23-be9f-4f8ce0c35647-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s\" (UID: \"116470cb-2dae-4b23-be9f-4f8ce0c35647\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s" Jul 25 01:16:23.477977 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:23.477931 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/116470cb-2dae-4b23-be9f-4f8ce0c35647-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s\" (UID: \"116470cb-2dae-4b23-be9f-4f8ce0c35647\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s" Jul 25 01:16:23.477977 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:23.477971 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4whlq\" (UniqueName: \"kubernetes.io/projected/116470cb-2dae-4b23-be9f-4f8ce0c35647-kube-api-access-4whlq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s\" (UID: \"116470cb-2dae-4b23-be9f-4f8ce0c35647\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s" Jul 25 01:16:23.578568 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:23.578483 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/116470cb-2dae-4b23-be9f-4f8ce0c35647-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s\" (UID: \"116470cb-2dae-4b23-be9f-4f8ce0c35647\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s" Jul 25 01:16:23.578568 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:23.578523 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4whlq\" (UniqueName: \"kubernetes.io/projected/116470cb-2dae-4b23-be9f-4f8ce0c35647-kube-api-access-4whlq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s\" (UID: \"116470cb-2dae-4b23-be9f-4f8ce0c35647\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s" Jul 25 01:16:23.578568 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:23.578558 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/116470cb-2dae-4b23-be9f-4f8ce0c35647-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s\" (UID: \"116470cb-2dae-4b23-be9f-4f8ce0c35647\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s" Jul 25 01:16:23.578977 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:23.578958 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/116470cb-2dae-4b23-be9f-4f8ce0c35647-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s\" (UID: \"116470cb-2dae-4b23-be9f-4f8ce0c35647\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s" Jul 25 01:16:23.579026 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:23.578967 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/116470cb-2dae-4b23-be9f-4f8ce0c35647-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s\" (UID: \"116470cb-2dae-4b23-be9f-4f8ce0c35647\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s" Jul 25 01:16:23.585882 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:23.585866 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4whlq\" (UniqueName: \"kubernetes.io/projected/116470cb-2dae-4b23-be9f-4f8ce0c35647-kube-api-access-4whlq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s\" (UID: \"116470cb-2dae-4b23-be9f-4f8ce0c35647\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s" Jul 25 01:16:23.598830 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:23.598796 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s" Jul 25 01:16:23.726401 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:23.726264 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s"] Jul 25 01:16:23.728934 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:16:23.728908 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116470cb_2dae_4b23_be9f_4f8ce0c35647.slice/crio-dc0b39ffee26b9767de25d4fe4e168de5fa9bfc2c8e70f2263363dcde5e441d4 WatchSource:0}: Error finding container dc0b39ffee26b9767de25d4fe4e168de5fa9bfc2c8e70f2263363dcde5e441d4: Status 404 returned error can't find the container with id dc0b39ffee26b9767de25d4fe4e168de5fa9bfc2c8e70f2263363dcde5e441d4 Jul 25 01:16:24.622269 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:24.622238 2554 generic.go:356] "Generic (PLEG): container finished" podID="116470cb-2dae-4b23-be9f-4f8ce0c35647" containerID="04c7fc1b0693b0bd134f2201bddb1719c4ecd3ccab148408061aa7ec4628e117" exitCode=0 Jul 25 01:16:24.622606 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:24.622323 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s" event={"ID":"116470cb-2dae-4b23-be9f-4f8ce0c35647","Type":"ContainerDied","Data":"04c7fc1b0693b0bd134f2201bddb1719c4ecd3ccab148408061aa7ec4628e117"} Jul 25 01:16:24.622606 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:24.622367 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s" event={"ID":"116470cb-2dae-4b23-be9f-4f8ce0c35647","Type":"ContainerStarted","Data":"dc0b39ffee26b9767de25d4fe4e168de5fa9bfc2c8e70f2263363dcde5e441d4"} Jul 25 01:16:25.628403 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:25.628367 2554 generic.go:356] "Generic (PLEG): container finished" podID="116470cb-2dae-4b23-be9f-4f8ce0c35647" containerID="8679e2178afc668f20e23a4db70d6a95cebc3430fb1269e38444641bbcb8d3e5" exitCode=0 Jul 25 01:16:25.628836 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:25.628455 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s" event={"ID":"116470cb-2dae-4b23-be9f-4f8ce0c35647","Type":"ContainerDied","Data":"8679e2178afc668f20e23a4db70d6a95cebc3430fb1269e38444641bbcb8d3e5"} Jul 25 01:16:26.634725 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:26.634692 2554 generic.go:356] "Generic (PLEG): container finished" podID="116470cb-2dae-4b23-be9f-4f8ce0c35647" containerID="d027c4c940d54d59bf34202f5c7aca53826c56eeaf61631f7041d7ebfb0fc112" exitCode=0 Jul 25 01:16:26.635080 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:26.634767 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s" event={"ID":"116470cb-2dae-4b23-be9f-4f8ce0c35647","Type":"ContainerDied","Data":"d027c4c940d54d59bf34202f5c7aca53826c56eeaf61631f7041d7ebfb0fc112"} Jul 25 01:16:27.760270 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:27.760250 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s" Jul 25 01:16:27.915414 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:27.915392 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/116470cb-2dae-4b23-be9f-4f8ce0c35647-util\") pod \"116470cb-2dae-4b23-be9f-4f8ce0c35647\" (UID: \"116470cb-2dae-4b23-be9f-4f8ce0c35647\") " Jul 25 01:16:27.915546 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:27.915434 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/116470cb-2dae-4b23-be9f-4f8ce0c35647-bundle\") pod \"116470cb-2dae-4b23-be9f-4f8ce0c35647\" (UID: \"116470cb-2dae-4b23-be9f-4f8ce0c35647\") " Jul 25 01:16:27.915546 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:27.915478 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4whlq\" (UniqueName: \"kubernetes.io/projected/116470cb-2dae-4b23-be9f-4f8ce0c35647-kube-api-access-4whlq\") pod \"116470cb-2dae-4b23-be9f-4f8ce0c35647\" (UID: \"116470cb-2dae-4b23-be9f-4f8ce0c35647\") " Jul 25 01:16:27.916175 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:27.916147 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116470cb-2dae-4b23-be9f-4f8ce0c35647-bundle" (OuterVolumeSpecName: "bundle") pod "116470cb-2dae-4b23-be9f-4f8ce0c35647" (UID: "116470cb-2dae-4b23-be9f-4f8ce0c35647"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:16:27.917847 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:27.917824 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116470cb-2dae-4b23-be9f-4f8ce0c35647-kube-api-access-4whlq" (OuterVolumeSpecName: "kube-api-access-4whlq") pod "116470cb-2dae-4b23-be9f-4f8ce0c35647" (UID: "116470cb-2dae-4b23-be9f-4f8ce0c35647"). InnerVolumeSpecName "kube-api-access-4whlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:16:27.920616 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:27.920580 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116470cb-2dae-4b23-be9f-4f8ce0c35647-util" (OuterVolumeSpecName: "util") pod "116470cb-2dae-4b23-be9f-4f8ce0c35647" (UID: "116470cb-2dae-4b23-be9f-4f8ce0c35647"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:16:28.016250 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:28.016218 2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/116470cb-2dae-4b23-be9f-4f8ce0c35647-util\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:16:28.016250 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:28.016243 2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/116470cb-2dae-4b23-be9f-4f8ce0c35647-bundle\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:16:28.016250 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:28.016252 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4whlq\" (UniqueName: \"kubernetes.io/projected/116470cb-2dae-4b23-be9f-4f8ce0c35647-kube-api-access-4whlq\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:16:28.644199 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:28.644168 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s" Jul 25 01:16:28.644199 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:28.644181 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2rz9s" event={"ID":"116470cb-2dae-4b23-be9f-4f8ce0c35647","Type":"ContainerDied","Data":"dc0b39ffee26b9767de25d4fe4e168de5fa9bfc2c8e70f2263363dcde5e441d4"} Jul 25 01:16:28.644393 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:28.644219 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc0b39ffee26b9767de25d4fe4e168de5fa9bfc2c8e70f2263363dcde5e441d4" Jul 25 01:16:40.832463 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.832433 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj"] Jul 25 01:16:40.832880 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.832694 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="116470cb-2dae-4b23-be9f-4f8ce0c35647" containerName="util" Jul 25 01:16:40.832880 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.832705 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="116470cb-2dae-4b23-be9f-4f8ce0c35647" containerName="util" Jul 25 01:16:40.832880 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.832719 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="116470cb-2dae-4b23-be9f-4f8ce0c35647" containerName="extract" Jul 25 01:16:40.832880 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.832728 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="116470cb-2dae-4b23-be9f-4f8ce0c35647" containerName="extract" Jul 25 01:16:40.832880 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.832744 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="116470cb-2dae-4b23-be9f-4f8ce0c35647" containerName="pull" Jul 25 01:16:40.832880 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.832749 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="116470cb-2dae-4b23-be9f-4f8ce0c35647" containerName="pull" Jul 25 01:16:40.832880 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.832790 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="116470cb-2dae-4b23-be9f-4f8ce0c35647" containerName="extract" Jul 25 01:16:40.836875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.836855 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" Jul 25 01:16:40.839110 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.839082 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 25 01:16:40.839248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.839137 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 25 01:16:40.839248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.839140 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 25 01:16:40.839248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.839155 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lrsw7\"" Jul 25 01:16:40.839248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.839229 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 25 01:16:40.851599 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.851568 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj"] Jul 25 01:16:40.901910 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.901881 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b56fd0c4-c24a-4890-81a8-a56a2c2175aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9844cf678-g7bhj\" (UID: \"b56fd0c4-c24a-4890-81a8-a56a2c2175aa\") " pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" Jul 25 01:16:40.902024 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.901935 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b56fd0c4-c24a-4890-81a8-a56a2c2175aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-9844cf678-g7bhj\" (UID: \"b56fd0c4-c24a-4890-81a8-a56a2c2175aa\") " pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" Jul 25 01:16:40.902024 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.902004 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b56fd0c4-c24a-4890-81a8-a56a2c2175aa-manifests\") pod \"opendatahub-operator-controller-manager-9844cf678-g7bhj\" (UID: \"b56fd0c4-c24a-4890-81a8-a56a2c2175aa\") " pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" Jul 25 01:16:40.902104 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.902030 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrhzx\" (UniqueName: \"kubernetes.io/projected/b56fd0c4-c24a-4890-81a8-a56a2c2175aa-kube-api-access-qrhzx\") pod \"opendatahub-operator-controller-manager-9844cf678-g7bhj\" (UID: \"b56fd0c4-c24a-4890-81a8-a56a2c2175aa\") " pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" Jul 25 01:16:40.902104 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:40.902057 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b56fd0c4-c24a-4890-81a8-a56a2c2175aa-tmp\") pod \"opendatahub-operator-controller-manager-9844cf678-g7bhj\" (UID: \"b56fd0c4-c24a-4890-81a8-a56a2c2175aa\") " pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" Jul 25 01:16:41.003241 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:41.003204 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b56fd0c4-c24a-4890-81a8-a56a2c2175aa-tmp\") pod \"opendatahub-operator-controller-manager-9844cf678-g7bhj\" (UID: \"b56fd0c4-c24a-4890-81a8-a56a2c2175aa\") " pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" Jul 25 01:16:41.003384 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:41.003265 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b56fd0c4-c24a-4890-81a8-a56a2c2175aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9844cf678-g7bhj\" (UID: \"b56fd0c4-c24a-4890-81a8-a56a2c2175aa\") " pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" Jul 25 01:16:41.003384 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:41.003313 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b56fd0c4-c24a-4890-81a8-a56a2c2175aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-9844cf678-g7bhj\" (UID: \"b56fd0c4-c24a-4890-81a8-a56a2c2175aa\") " pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" Jul 25 01:16:41.003384 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:41.003354 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b56fd0c4-c24a-4890-81a8-a56a2c2175aa-manifests\") pod \"opendatahub-operator-controller-manager-9844cf678-g7bhj\" (UID: \"b56fd0c4-c24a-4890-81a8-a56a2c2175aa\") " pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" Jul 25 01:16:41.003542 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:41.003382 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrhzx\" (UniqueName: \"kubernetes.io/projected/b56fd0c4-c24a-4890-81a8-a56a2c2175aa-kube-api-access-qrhzx\") pod \"opendatahub-operator-controller-manager-9844cf678-g7bhj\" (UID: \"b56fd0c4-c24a-4890-81a8-a56a2c2175aa\") " pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" Jul 25 01:16:41.003758 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:41.003733 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b56fd0c4-c24a-4890-81a8-a56a2c2175aa-manifests\") pod \"opendatahub-operator-controller-manager-9844cf678-g7bhj\" (UID: \"b56fd0c4-c24a-4890-81a8-a56a2c2175aa\") " pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" Jul 25 01:16:41.005954 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:41.005929 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b56fd0c4-c24a-4890-81a8-a56a2c2175aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9844cf678-g7bhj\" (UID: \"b56fd0c4-c24a-4890-81a8-a56a2c2175aa\") " pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" Jul 25 01:16:41.006179 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:41.006160 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b56fd0c4-c24a-4890-81a8-a56a2c2175aa-tmp\") pod \"opendatahub-operator-controller-manager-9844cf678-g7bhj\" (UID: \"b56fd0c4-c24a-4890-81a8-a56a2c2175aa\") " pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" Jul 25 01:16:41.006319 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:41.006302 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b56fd0c4-c24a-4890-81a8-a56a2c2175aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-9844cf678-g7bhj\" (UID: \"b56fd0c4-c24a-4890-81a8-a56a2c2175aa\") " pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" Jul 25 01:16:41.010943 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:41.010922 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrhzx\" (UniqueName: \"kubernetes.io/projected/b56fd0c4-c24a-4890-81a8-a56a2c2175aa-kube-api-access-qrhzx\") pod \"opendatahub-operator-controller-manager-9844cf678-g7bhj\" (UID: \"b56fd0c4-c24a-4890-81a8-a56a2c2175aa\") " pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" Jul 25 01:16:41.145958 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:41.145877 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" Jul 25 01:16:41.278554 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:41.278515 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj"] Jul 25 01:16:41.282626 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:16:41.282599 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56fd0c4_c24a_4890_81a8_a56a2c2175aa.slice/crio-ed46c9aa04dc9b382abbdb69c62ded1becaf71caca682e61d47f2a23cd2966a1 WatchSource:0}: Error finding container ed46c9aa04dc9b382abbdb69c62ded1becaf71caca682e61d47f2a23cd2966a1: Status 404 returned error can't find the container with id ed46c9aa04dc9b382abbdb69c62ded1becaf71caca682e61d47f2a23cd2966a1 Jul 25 01:16:41.692553 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:41.692522 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" event={"ID":"b56fd0c4-c24a-4890-81a8-a56a2c2175aa","Type":"ContainerStarted","Data":"ed46c9aa04dc9b382abbdb69c62ded1becaf71caca682e61d47f2a23cd2966a1"} Jul 25 01:16:44.712243 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:44.712205 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" event={"ID":"b56fd0c4-c24a-4890-81a8-a56a2c2175aa","Type":"ContainerStarted","Data":"ff716b2d7b10bff559ac89993d8432ecd749d97a1b3bd50f48a3b9c328d6dbb7"} Jul 25 01:16:45.717710 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:45.717681 2554 generic.go:356] "Generic (PLEG): container finished" podID="b56fd0c4-c24a-4890-81a8-a56a2c2175aa" containerID="ff716b2d7b10bff559ac89993d8432ecd749d97a1b3bd50f48a3b9c328d6dbb7" exitCode=0 Jul 25 01:16:45.718111 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:45.717743 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" event={"ID":"b56fd0c4-c24a-4890-81a8-a56a2c2175aa","Type":"ContainerDied","Data":"ff716b2d7b10bff559ac89993d8432ecd749d97a1b3bd50f48a3b9c328d6dbb7"} Jul 25 01:16:46.723533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:46.723500 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" event={"ID":"b56fd0c4-c24a-4890-81a8-a56a2c2175aa","Type":"ContainerStarted","Data":"44e8800db567fc9d685b46a06b44104e28d98b81caf2fb69bff67e318c7bc6e9"} Jul 25 01:16:46.723944 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:46.723618 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" Jul 25 01:16:46.744024 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:46.743982 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" podStartSLOduration=2.345322577 podStartE2EDuration="6.74396817s" totalImagesPullingTime="3.355814068s" totalInitContainerRuntime="1.042831525s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:16:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:16:46.742624414 +0000 UTC m=+347.864110260" watchObservedRunningTime="2026-07-25 01:16:46.74396817 +0000 UTC m=+347.865454019" Jul 25 01:16:57.730418 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:16:57.730333 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9844cf678-g7bhj" Jul 25 01:17:00.510637 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:00.510604 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6bd6744b-hpshk"] Jul 25 01:17:00.516374 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:00.516350 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c6bd6744b-hpshk" Jul 25 01:17:00.519494 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:00.519474 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 25 01:17:00.519942 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:00.519925 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 25 01:17:00.520327 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:00.520306 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pzbvk\"" Jul 25 01:17:00.534369 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:00.534349 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6bd6744b-hpshk"] Jul 25 01:17:00.647452 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:00.647425 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vffh\" (UniqueName: \"kubernetes.io/projected/f5b1f7e4-edc6-4178-a97e-418789d1fafd-kube-api-access-6vffh\") pod \"kube-auth-proxy-7c6bd6744b-hpshk\" (UID: \"f5b1f7e4-edc6-4178-a97e-418789d1fafd\") " pod="openshift-ingress/kube-auth-proxy-7c6bd6744b-hpshk" Jul 25 01:17:00.647624 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:00.647462 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5b1f7e4-edc6-4178-a97e-418789d1fafd-tls-certs\") pod \"kube-auth-proxy-7c6bd6744b-hpshk\" (UID: \"f5b1f7e4-edc6-4178-a97e-418789d1fafd\") " pod="openshift-ingress/kube-auth-proxy-7c6bd6744b-hpshk" Jul 25 01:17:00.647624 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:00.647483 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5b1f7e4-edc6-4178-a97e-418789d1fafd-tmp\") pod \"kube-auth-proxy-7c6bd6744b-hpshk\" (UID: \"f5b1f7e4-edc6-4178-a97e-418789d1fafd\") " pod="openshift-ingress/kube-auth-proxy-7c6bd6744b-hpshk" Jul 25 01:17:00.747939 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:00.747909 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vffh\" (UniqueName: \"kubernetes.io/projected/f5b1f7e4-edc6-4178-a97e-418789d1fafd-kube-api-access-6vffh\") pod \"kube-auth-proxy-7c6bd6744b-hpshk\" (UID: \"f5b1f7e4-edc6-4178-a97e-418789d1fafd\") " pod="openshift-ingress/kube-auth-proxy-7c6bd6744b-hpshk" Jul 25 01:17:00.748118 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:00.747950 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5b1f7e4-edc6-4178-a97e-418789d1fafd-tls-certs\") pod \"kube-auth-proxy-7c6bd6744b-hpshk\" (UID: \"f5b1f7e4-edc6-4178-a97e-418789d1fafd\") " pod="openshift-ingress/kube-auth-proxy-7c6bd6744b-hpshk" Jul 25 01:17:00.748118 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:00.747967 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5b1f7e4-edc6-4178-a97e-418789d1fafd-tmp\") pod \"kube-auth-proxy-7c6bd6744b-hpshk\" (UID: \"f5b1f7e4-edc6-4178-a97e-418789d1fafd\") " pod="openshift-ingress/kube-auth-proxy-7c6bd6744b-hpshk" Jul 25 01:17:00.750262 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:00.750238 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5b1f7e4-edc6-4178-a97e-418789d1fafd-tmp\") pod \"kube-auth-proxy-7c6bd6744b-hpshk\" (UID: \"f5b1f7e4-edc6-4178-a97e-418789d1fafd\") " pod="openshift-ingress/kube-auth-proxy-7c6bd6744b-hpshk" Jul 25 01:17:00.750488 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:00.750467 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5b1f7e4-edc6-4178-a97e-418789d1fafd-tls-certs\") pod \"kube-auth-proxy-7c6bd6744b-hpshk\" (UID: \"f5b1f7e4-edc6-4178-a97e-418789d1fafd\") " pod="openshift-ingress/kube-auth-proxy-7c6bd6744b-hpshk" Jul 25 01:17:00.756291 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:00.756269 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vffh\" (UniqueName: \"kubernetes.io/projected/f5b1f7e4-edc6-4178-a97e-418789d1fafd-kube-api-access-6vffh\") pod \"kube-auth-proxy-7c6bd6744b-hpshk\" (UID: \"f5b1f7e4-edc6-4178-a97e-418789d1fafd\") " pod="openshift-ingress/kube-auth-proxy-7c6bd6744b-hpshk" Jul 25 01:17:00.825649 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:00.825579 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c6bd6744b-hpshk" Jul 25 01:17:00.961254 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:00.961226 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6bd6744b-hpshk"] Jul 25 01:17:00.964085 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:17:00.964058 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b1f7e4_edc6_4178_a97e_418789d1fafd.slice/crio-e2c334557e7e26273be688b90a784f89f1bac30aaa84eef4061764bf6aee3c37 WatchSource:0}: Error finding container e2c334557e7e26273be688b90a784f89f1bac30aaa84eef4061764bf6aee3c37: Status 404 returned error can't find the container with id e2c334557e7e26273be688b90a784f89f1bac30aaa84eef4061764bf6aee3c37 Jul 25 01:17:01.606706 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.606659 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8"] Jul 25 01:17:01.610156 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.610130 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.612508 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.612484 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 25 01:17:01.613029 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.612839 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 25 01:17:01.613029 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.612844 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 25 01:17:01.613029 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.612847 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 25 01:17:01.613375 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.613359 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qkm6g\"" Jul 25 01:17:01.626030 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.626009 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8"] Jul 25 01:17:01.655482 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.655451 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21707576-91dd-4070-889b-8e5704b44a0f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.655623 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.655497 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21707576-91dd-4070-889b-8e5704b44a0f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.655623 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.655543 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21707576-91dd-4070-889b-8e5704b44a0f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.655623 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.655588 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21707576-91dd-4070-889b-8e5704b44a0f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.655767 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.655659 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21707576-91dd-4070-889b-8e5704b44a0f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.655767 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.655683 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw5qn\" (UniqueName: \"kubernetes.io/projected/21707576-91dd-4070-889b-8e5704b44a0f-kube-api-access-gw5qn\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.655767 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.655711 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21707576-91dd-4070-889b-8e5704b44a0f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.757013 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.756981 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21707576-91dd-4070-889b-8e5704b44a0f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.757184 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.757026 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21707576-91dd-4070-889b-8e5704b44a0f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.757184 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.757049 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21707576-91dd-4070-889b-8e5704b44a0f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.757184 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.757089 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21707576-91dd-4070-889b-8e5704b44a0f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.757184 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.757105 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw5qn\" (UniqueName: \"kubernetes.io/projected/21707576-91dd-4070-889b-8e5704b44a0f-kube-api-access-gw5qn\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.757184 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.757131 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21707576-91dd-4070-889b-8e5704b44a0f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.757184 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.757181 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21707576-91dd-4070-889b-8e5704b44a0f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.757717 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.757689 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21707576-91dd-4070-889b-8e5704b44a0f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.760066 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.760042 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21707576-91dd-4070-889b-8e5704b44a0f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.760066 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.760051 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21707576-91dd-4070-889b-8e5704b44a0f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.760190 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.760166 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21707576-91dd-4070-889b-8e5704b44a0f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.760264 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.760247 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21707576-91dd-4070-889b-8e5704b44a0f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.767779 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.767759 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21707576-91dd-4070-889b-8e5704b44a0f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.767895 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.767813 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw5qn\" (UniqueName: \"kubernetes.io/projected/21707576-91dd-4070-889b-8e5704b44a0f-kube-api-access-gw5qn\") pod \"istiod-openshift-gateway-7b7995d5cd-qwcr8\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:01.784274 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.784244 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c6bd6744b-hpshk" event={"ID":"f5b1f7e4-edc6-4178-a97e-418789d1fafd","Type":"ContainerStarted","Data":"e2c334557e7e26273be688b90a784f89f1bac30aaa84eef4061764bf6aee3c37"} Jul 25 01:17:01.922159 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:01.922126 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:02.060584 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:02.060562 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8"] Jul 25 01:17:02.062664 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:17:02.062633 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21707576_91dd_4070_889b_8e5704b44a0f.slice/crio-287d1fe900bebbd797f839ac53935302fb641da5129ebe1b1f6b26c9de15923f WatchSource:0}: Error finding container 287d1fe900bebbd797f839ac53935302fb641da5129ebe1b1f6b26c9de15923f: Status 404 returned error can't find the container with id 287d1fe900bebbd797f839ac53935302fb641da5129ebe1b1f6b26c9de15923f Jul 25 01:17:02.789741 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:02.789706 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" event={"ID":"21707576-91dd-4070-889b-8e5704b44a0f","Type":"ContainerStarted","Data":"287d1fe900bebbd797f839ac53935302fb641da5129ebe1b1f6b26c9de15923f"} Jul 25 01:17:03.796353 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:03.796313 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c6bd6744b-hpshk" event={"ID":"f5b1f7e4-edc6-4178-a97e-418789d1fafd","Type":"ContainerStarted","Data":"abfb49943b427038abf708207876f1b24583f971768b5f37cb9e63a54ae0e784"} Jul 25 01:17:03.812548 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:03.812484 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c6bd6744b-hpshk" podStartSLOduration=1.17397985 podStartE2EDuration="3.812465522s" totalImagesPullingTime="2.638485672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:17:03.810649008 +0000 UTC m=+364.932134860" watchObservedRunningTime="2026-07-25 01:17:03.812465522 +0000 UTC m=+364.933951373" Jul 25 01:17:05.989714 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:05.989673 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 25 01:17:05.990017 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:05.989747 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 25 01:17:06.812957 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:06.812921 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" event={"ID":"21707576-91dd-4070-889b-8e5704b44a0f","Type":"ContainerStarted","Data":"fe381977fc7cbbb16212d6eb426061854fd4e439e5c1e43db0446be0fe2968d3"} Jul 25 01:17:06.813111 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:06.812991 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:06.842533 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:06.842488 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" podStartSLOduration=1.9177059650000001 podStartE2EDuration="5.842474399s" totalImagesPullingTime="3.924768434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:17:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:17:06.841161828 +0000 UTC m=+367.962647678" watchObservedRunningTime="2026-07-25 01:17:06.842474399 +0000 UTC m=+367.963960248" Jul 25 01:17:07.819675 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:07.819647 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:11.350214 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.350180 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-s7bpn"] Jul 25 01:17:11.353411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.353392 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s7bpn" Jul 25 01:17:11.355658 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.355640 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 25 01:17:11.355658 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.355647 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-6lsrf\"" Jul 25 01:17:11.361300 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.361274 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-nqfl4"] Jul 25 01:17:11.365367 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.365345 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-s7bpn"] Jul 25 01:17:11.365471 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.365429 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nqfl4" Jul 25 01:17:11.367698 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.367680 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 25 01:17:11.368088 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.368068 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-dltwd\"" Jul 25 01:17:11.374579 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.374561 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-nqfl4"] Jul 25 01:17:11.439324 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.439294 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6255dbe5-ec40-45da-a62e-ad6ea8fba5f9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-nqfl4\" (UID: \"6255dbe5-ec40-45da-a62e-ad6ea8fba5f9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nqfl4" Jul 25 01:17:11.439477 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.439434 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd9qz\" (UniqueName: \"kubernetes.io/projected/6255dbe5-ec40-45da-a62e-ad6ea8fba5f9-kube-api-access-rd9qz\") pod \"llmisvc-controller-manager-777c6bd7d5-nqfl4\" (UID: \"6255dbe5-ec40-45da-a62e-ad6ea8fba5f9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nqfl4" Jul 25 01:17:11.439600 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.439575 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8fec9f9-14dc-4a9a-a56a-169a655d827d-cert\") pod \"kserve-controller-manager-d6f4b65b8-s7bpn\" (UID: \"f8fec9f9-14dc-4a9a-a56a-169a655d827d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s7bpn" Jul 25 01:17:11.439790 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.439748 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg7vq\" (UniqueName: \"kubernetes.io/projected/f8fec9f9-14dc-4a9a-a56a-169a655d827d-kube-api-access-wg7vq\") pod \"kserve-controller-manager-d6f4b65b8-s7bpn\" (UID: \"f8fec9f9-14dc-4a9a-a56a-169a655d827d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s7bpn" Jul 25 01:17:11.541189 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.541155 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd9qz\" (UniqueName: \"kubernetes.io/projected/6255dbe5-ec40-45da-a62e-ad6ea8fba5f9-kube-api-access-rd9qz\") pod \"llmisvc-controller-manager-777c6bd7d5-nqfl4\" (UID: \"6255dbe5-ec40-45da-a62e-ad6ea8fba5f9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nqfl4" Jul 25 01:17:11.541356 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.541211 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8fec9f9-14dc-4a9a-a56a-169a655d827d-cert\") pod \"kserve-controller-manager-d6f4b65b8-s7bpn\" (UID: \"f8fec9f9-14dc-4a9a-a56a-169a655d827d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s7bpn" Jul 25 01:17:11.541356 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.541241 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg7vq\" (UniqueName: \"kubernetes.io/projected/f8fec9f9-14dc-4a9a-a56a-169a655d827d-kube-api-access-wg7vq\") pod \"kserve-controller-manager-d6f4b65b8-s7bpn\" (UID: \"f8fec9f9-14dc-4a9a-a56a-169a655d827d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s7bpn" Jul 25 01:17:11.541356 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.541283 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6255dbe5-ec40-45da-a62e-ad6ea8fba5f9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-nqfl4\" (UID: \"6255dbe5-ec40-45da-a62e-ad6ea8fba5f9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nqfl4" Jul 25 01:17:11.543783 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.543752 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8fec9f9-14dc-4a9a-a56a-169a655d827d-cert\") pod \"kserve-controller-manager-d6f4b65b8-s7bpn\" (UID: \"f8fec9f9-14dc-4a9a-a56a-169a655d827d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s7bpn" Jul 25 01:17:11.543925 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.543847 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6255dbe5-ec40-45da-a62e-ad6ea8fba5f9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-nqfl4\" (UID: \"6255dbe5-ec40-45da-a62e-ad6ea8fba5f9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nqfl4" Jul 25 01:17:11.550005 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.549979 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg7vq\" (UniqueName: \"kubernetes.io/projected/f8fec9f9-14dc-4a9a-a56a-169a655d827d-kube-api-access-wg7vq\") pod \"kserve-controller-manager-d6f4b65b8-s7bpn\" (UID: \"f8fec9f9-14dc-4a9a-a56a-169a655d827d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s7bpn" Jul 25 01:17:11.550122 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.550107 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd9qz\" (UniqueName: \"kubernetes.io/projected/6255dbe5-ec40-45da-a62e-ad6ea8fba5f9-kube-api-access-rd9qz\") pod \"llmisvc-controller-manager-777c6bd7d5-nqfl4\" (UID: \"6255dbe5-ec40-45da-a62e-ad6ea8fba5f9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nqfl4" Jul 25 01:17:11.667701 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.667676 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s7bpn" Jul 25 01:17:11.677474 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.677446 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nqfl4" Jul 25 01:17:11.817066 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.817040 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-s7bpn"] Jul 25 01:17:11.819374 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:17:11.819352 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8fec9f9_14dc_4a9a_a56a_169a655d827d.slice/crio-56e83b0f6f90130f64711e2fe812ceae83d57927cac9f276d4a875d64963c1a6 WatchSource:0}: Error finding container 56e83b0f6f90130f64711e2fe812ceae83d57927cac9f276d4a875d64963c1a6: Status 404 returned error can't find the container with id 56e83b0f6f90130f64711e2fe812ceae83d57927cac9f276d4a875d64963c1a6 Jul 25 01:17:11.833170 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.833145 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-nqfl4"] Jul 25 01:17:11.835681 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:17:11.835660 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6255dbe5_ec40_45da_a62e_ad6ea8fba5f9.slice/crio-29fa2b11e605ec225b11fd67f83f1519754e2e1e3ea472845b0c67cc879422b4 WatchSource:0}: Error finding container 29fa2b11e605ec225b11fd67f83f1519754e2e1e3ea472845b0c67cc879422b4: Status 404 returned error can't find the container with id 29fa2b11e605ec225b11fd67f83f1519754e2e1e3ea472845b0c67cc879422b4 Jul 25 01:17:11.836584 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:11.836557 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s7bpn" event={"ID":"f8fec9f9-14dc-4a9a-a56a-169a655d827d","Type":"ContainerStarted","Data":"56e83b0f6f90130f64711e2fe812ceae83d57927cac9f276d4a875d64963c1a6"} Jul 25 01:17:12.846297 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:12.846226 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nqfl4" event={"ID":"6255dbe5-ec40-45da-a62e-ad6ea8fba5f9","Type":"ContainerStarted","Data":"29fa2b11e605ec225b11fd67f83f1519754e2e1e3ea472845b0c67cc879422b4"} Jul 25 01:17:13.536398 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:13.536354 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8"] Jul 25 01:17:13.536580 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:13.536544 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" podUID="21707576-91dd-4070-889b-8e5704b44a0f" containerName="discovery" containerID="cri-o://fe381977fc7cbbb16212d6eb426061854fd4e439e5c1e43db0446be0fe2968d3" gracePeriod=30 Jul 25 01:17:13.855028 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:13.854943 2554 generic.go:356] "Generic (PLEG): container finished" podID="21707576-91dd-4070-889b-8e5704b44a0f" containerID="fe381977fc7cbbb16212d6eb426061854fd4e439e5c1e43db0446be0fe2968d3" exitCode=0 Jul 25 01:17:13.855411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:13.855028 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" event={"ID":"21707576-91dd-4070-889b-8e5704b44a0f","Type":"ContainerDied","Data":"fe381977fc7cbbb16212d6eb426061854fd4e439e5c1e43db0446be0fe2968d3"} Jul 25 01:17:14.320936 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.320907 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:14.467126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.467091 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21707576-91dd-4070-889b-8e5704b44a0f-istio-token\") pod \"21707576-91dd-4070-889b-8e5704b44a0f\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " Jul 25 01:17:14.467126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.467140 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gw5qn\" (UniqueName: \"kubernetes.io/projected/21707576-91dd-4070-889b-8e5704b44a0f-kube-api-access-gw5qn\") pod \"21707576-91dd-4070-889b-8e5704b44a0f\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " Jul 25 01:17:14.467383 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.467183 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21707576-91dd-4070-889b-8e5704b44a0f-istio-kubeconfig\") pod \"21707576-91dd-4070-889b-8e5704b44a0f\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " Jul 25 01:17:14.467383 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.467285 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21707576-91dd-4070-889b-8e5704b44a0f-istio-csr-dns-cert\") pod \"21707576-91dd-4070-889b-8e5704b44a0f\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " Jul 25 01:17:14.467383 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.467330 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21707576-91dd-4070-889b-8e5704b44a0f-local-certs\") pod \"21707576-91dd-4070-889b-8e5704b44a0f\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " Jul 25 01:17:14.467531 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.467421 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21707576-91dd-4070-889b-8e5704b44a0f-cacerts\") pod \"21707576-91dd-4070-889b-8e5704b44a0f\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " Jul 25 01:17:14.467531 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.467453 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21707576-91dd-4070-889b-8e5704b44a0f-istio-csr-ca-configmap\") pod \"21707576-91dd-4070-889b-8e5704b44a0f\" (UID: \"21707576-91dd-4070-889b-8e5704b44a0f\") " Jul 25 01:17:14.468050 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.468003 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21707576-91dd-4070-889b-8e5704b44a0f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "21707576-91dd-4070-889b-8e5704b44a0f" (UID: "21707576-91dd-4070-889b-8e5704b44a0f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 01:17:14.470850 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.470790 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21707576-91dd-4070-889b-8e5704b44a0f-cacerts" (OuterVolumeSpecName: "cacerts") pod "21707576-91dd-4070-889b-8e5704b44a0f" (UID: "21707576-91dd-4070-889b-8e5704b44a0f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 01:17:14.470985 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.470896 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21707576-91dd-4070-889b-8e5704b44a0f-local-certs" (OuterVolumeSpecName: "local-certs") pod "21707576-91dd-4070-889b-8e5704b44a0f" (UID: "21707576-91dd-4070-889b-8e5704b44a0f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:17:14.470985 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.470844 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21707576-91dd-4070-889b-8e5704b44a0f-istio-token" (OuterVolumeSpecName: "istio-token") pod "21707576-91dd-4070-889b-8e5704b44a0f" (UID: "21707576-91dd-4070-889b-8e5704b44a0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:17:14.471251 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.471212 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21707576-91dd-4070-889b-8e5704b44a0f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "21707576-91dd-4070-889b-8e5704b44a0f" (UID: "21707576-91dd-4070-889b-8e5704b44a0f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 01:17:14.471367 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.471345 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21707576-91dd-4070-889b-8e5704b44a0f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "21707576-91dd-4070-889b-8e5704b44a0f" (UID: "21707576-91dd-4070-889b-8e5704b44a0f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 01:17:14.471465 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.471443 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21707576-91dd-4070-889b-8e5704b44a0f-kube-api-access-gw5qn" (OuterVolumeSpecName: "kube-api-access-gw5qn") pod "21707576-91dd-4070-889b-8e5704b44a0f" (UID: "21707576-91dd-4070-889b-8e5704b44a0f"). InnerVolumeSpecName "kube-api-access-gw5qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:17:14.568573 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.568482 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw5qn\" (UniqueName: \"kubernetes.io/projected/21707576-91dd-4070-889b-8e5704b44a0f-kube-api-access-gw5qn\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:17:14.568573 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.568513 2554 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21707576-91dd-4070-889b-8e5704b44a0f-istio-kubeconfig\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:17:14.568573 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.568527 2554 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21707576-91dd-4070-889b-8e5704b44a0f-istio-csr-dns-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:17:14.568573 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.568540 2554 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21707576-91dd-4070-889b-8e5704b44a0f-local-certs\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:17:14.568573 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.568551 2554 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21707576-91dd-4070-889b-8e5704b44a0f-cacerts\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:17:14.568573 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.568563 2554 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21707576-91dd-4070-889b-8e5704b44a0f-istio-csr-ca-configmap\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:17:14.568573 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.568576 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21707576-91dd-4070-889b-8e5704b44a0f-istio-token\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:17:14.860441 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.860379 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" Jul 25 01:17:14.860441 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.860393 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" event={"ID":"21707576-91dd-4070-889b-8e5704b44a0f","Type":"ContainerDied","Data":"287d1fe900bebbd797f839ac53935302fb641da5129ebe1b1f6b26c9de15923f"} Jul 25 01:17:14.861011 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.860449 2554 scope.go:117] "RemoveContainer" containerID="fe381977fc7cbbb16212d6eb426061854fd4e439e5c1e43db0446be0fe2968d3" Jul 25 01:17:14.883516 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.883492 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8"] Jul 25 01:17:14.887149 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:14.887127 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8"] Jul 25 01:17:15.380386 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:15.380357 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21707576-91dd-4070-889b-8e5704b44a0f" path="/var/lib/kubelet/pods/21707576-91dd-4070-889b-8e5704b44a0f/volumes" Jul 25 01:17:15.866292 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:15.866259 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nqfl4" event={"ID":"6255dbe5-ec40-45da-a62e-ad6ea8fba5f9","Type":"ContainerStarted","Data":"d61f73dcfce300563cbbe27a4801447a907c314044f0dcb3f4d27e76ef21d950"} Jul 25 01:17:15.866714 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:15.866341 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nqfl4" Jul 25 01:17:15.868540 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:15.868515 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s7bpn" event={"ID":"f8fec9f9-14dc-4a9a-a56a-169a655d827d","Type":"ContainerStarted","Data":"69dc307883f91e25a89da35a1cc5994ee4723d1aed6d2d5c3f9d53f8bd0e3ac6"} Jul 25 01:17:15.868665 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:15.868566 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s7bpn" Jul 25 01:17:15.885369 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:15.885317 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nqfl4" podStartSLOduration=1.790426955 podStartE2EDuration="4.88530385s" totalImagesPullingTime="3.094876895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:17:15.884447747 +0000 UTC m=+377.005933598" watchObservedRunningTime="2026-07-25 01:17:15.88530385 +0000 UTC m=+377.006789678" Jul 25 01:17:15.900075 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:15.900036 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s7bpn" podStartSLOduration=1.793473835 podStartE2EDuration="4.900022215s" totalImagesPullingTime="3.10654838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:17:15.899436243 +0000 UTC m=+377.020922093" watchObservedRunningTime="2026-07-25 01:17:15.900022215 +0000 UTC m=+377.021508066" Jul 25 01:17:18.819419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:18.819376 2554 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-qwcr8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.24:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 25 01:17:18.819844 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:18.819445 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qwcr8" podUID="21707576-91dd-4070-889b-8e5704b44a0f" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.24:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 25 01:17:46.876222 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:46.876195 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nqfl4" Jul 25 01:17:46.879104 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:17:46.879084 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s7bpn" Jul 25 01:18:01.453370 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:01.453341 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vnq2h"] Jul 25 01:18:01.453742 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:01.453614 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21707576-91dd-4070-889b-8e5704b44a0f" containerName="discovery" Jul 25 01:18:01.453742 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:01.453624 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="21707576-91dd-4070-889b-8e5704b44a0f" containerName="discovery" Jul 25 01:18:01.453742 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:01.453669 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="21707576-91dd-4070-889b-8e5704b44a0f" containerName="discovery" Jul 25 01:18:01.457197 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:01.457170 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vnq2h" Jul 25 01:18:01.459319 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:01.459294 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 25 01:18:01.459453 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:01.459294 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 25 01:18:01.460077 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:01.460054 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-vdzjw\"" Jul 25 01:18:01.466226 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:01.465750 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vnq2h"] Jul 25 01:18:01.524443 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:01.524416 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crzrz\" (UniqueName: \"kubernetes.io/projected/99740d03-3566-4c91-95e9-c78b6cab77b6-kube-api-access-crzrz\") pod \"kuadrant-operator-catalog-vnq2h\" (UID: \"99740d03-3566-4c91-95e9-c78b6cab77b6\") " pod="kuadrant-system/kuadrant-operator-catalog-vnq2h" Jul 25 01:18:01.624710 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:01.624684 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crzrz\" (UniqueName: \"kubernetes.io/projected/99740d03-3566-4c91-95e9-c78b6cab77b6-kube-api-access-crzrz\") pod \"kuadrant-operator-catalog-vnq2h\" (UID: \"99740d03-3566-4c91-95e9-c78b6cab77b6\") " pod="kuadrant-system/kuadrant-operator-catalog-vnq2h" Jul 25 01:18:01.635224 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:01.635197 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crzrz\" (UniqueName: \"kubernetes.io/projected/99740d03-3566-4c91-95e9-c78b6cab77b6-kube-api-access-crzrz\") pod \"kuadrant-operator-catalog-vnq2h\" (UID: \"99740d03-3566-4c91-95e9-c78b6cab77b6\") " pod="kuadrant-system/kuadrant-operator-catalog-vnq2h" Jul 25 01:18:01.769070 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:01.768983 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vnq2h" Jul 25 01:18:01.897904 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:01.897868 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vnq2h"] Jul 25 01:18:01.900948 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:18:01.900909 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99740d03_3566_4c91_95e9_c78b6cab77b6.slice/crio-8f72c419f969b5e277009fa34eca9dd4164575cf128aa833d01a4aa09f86e5d0 WatchSource:0}: Error finding container 8f72c419f969b5e277009fa34eca9dd4164575cf128aa833d01a4aa09f86e5d0: Status 404 returned error can't find the container with id 8f72c419f969b5e277009fa34eca9dd4164575cf128aa833d01a4aa09f86e5d0 Jul 25 01:18:01.975388 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:01.975361 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vnq2h"] Jul 25 01:18:02.004790 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:02.004766 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8zlzc"] Jul 25 01:18:02.008510 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:02.008496 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8zlzc" Jul 25 01:18:02.016523 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:02.016496 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8zlzc"] Jul 25 01:18:02.061462 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:02.061397 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vnq2h" event={"ID":"99740d03-3566-4c91-95e9-c78b6cab77b6","Type":"ContainerStarted","Data":"8f72c419f969b5e277009fa34eca9dd4164575cf128aa833d01a4aa09f86e5d0"} Jul 25 01:18:02.129884 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:02.129859 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cv4g\" (UniqueName: \"kubernetes.io/projected/0f0054b1-5908-4314-b941-925a25ef9c68-kube-api-access-4cv4g\") pod \"kuadrant-operator-catalog-8zlzc\" (UID: \"0f0054b1-5908-4314-b941-925a25ef9c68\") " pod="kuadrant-system/kuadrant-operator-catalog-8zlzc" Jul 25 01:18:02.231329 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:02.231299 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cv4g\" (UniqueName: \"kubernetes.io/projected/0f0054b1-5908-4314-b941-925a25ef9c68-kube-api-access-4cv4g\") pod \"kuadrant-operator-catalog-8zlzc\" (UID: \"0f0054b1-5908-4314-b941-925a25ef9c68\") " pod="kuadrant-system/kuadrant-operator-catalog-8zlzc" Jul 25 01:18:02.242899 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:02.242878 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cv4g\" (UniqueName: \"kubernetes.io/projected/0f0054b1-5908-4314-b941-925a25ef9c68-kube-api-access-4cv4g\") pod \"kuadrant-operator-catalog-8zlzc\" (UID: \"0f0054b1-5908-4314-b941-925a25ef9c68\") " pod="kuadrant-system/kuadrant-operator-catalog-8zlzc" Jul 25 01:18:02.317923 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:02.317869 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8zlzc" Jul 25 01:18:02.447436 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:02.447415 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8zlzc"] Jul 25 01:18:02.449851 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:18:02.449823 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0054b1_5908_4314_b941_925a25ef9c68.slice/crio-1f1bf5808ecebdb089bbc87b4c341548100316fb04d327d3f8339d97952c169c WatchSource:0}: Error finding container 1f1bf5808ecebdb089bbc87b4c341548100316fb04d327d3f8339d97952c169c: Status 404 returned error can't find the container with id 1f1bf5808ecebdb089bbc87b4c341548100316fb04d327d3f8339d97952c169c Jul 25 01:18:03.069795 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:03.069754 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8zlzc" event={"ID":"0f0054b1-5908-4314-b941-925a25ef9c68","Type":"ContainerStarted","Data":"1f1bf5808ecebdb089bbc87b4c341548100316fb04d327d3f8339d97952c169c"} Jul 25 01:18:05.079635 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:05.079596 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8zlzc" event={"ID":"0f0054b1-5908-4314-b941-925a25ef9c68","Type":"ContainerStarted","Data":"2e6b9f3e5a355272b4e954c38bd2879d0eb8b5eb59bc5ef044d28b709c4a2cba"} Jul 25 01:18:05.081226 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:05.081201 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vnq2h" event={"ID":"99740d03-3566-4c91-95e9-c78b6cab77b6","Type":"ContainerStarted","Data":"e9e6b0b7e49dd43f1618324e9d2865cca3d946d6a3adb21931372918daefb748"} Jul 25 01:18:05.081320 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:05.081257 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-vnq2h" podUID="99740d03-3566-4c91-95e9-c78b6cab77b6" containerName="registry-server" containerID="cri-o://e9e6b0b7e49dd43f1618324e9d2865cca3d946d6a3adb21931372918daefb748" gracePeriod=2 Jul 25 01:18:05.094252 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:05.094208 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-8zlzc" podStartSLOduration=2.249750349 podStartE2EDuration="4.09419295s" totalImagesPullingTime="1.844442601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:18:05.093103928 +0000 UTC m=+426.214589777" watchObservedRunningTime="2026-07-25 01:18:05.09419295 +0000 UTC m=+426.215678825" Jul 25 01:18:05.107301 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:05.107259 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-vnq2h" podStartSLOduration=1.7173464699999998 podStartE2EDuration="4.107246731s" totalImagesPullingTime="2.389900261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:18:05.105871246 +0000 UTC m=+426.227357096" watchObservedRunningTime="2026-07-25 01:18:05.107246731 +0000 UTC m=+426.228732580" Jul 25 01:18:05.329157 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:05.329132 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vnq2h" Jul 25 01:18:05.457767 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:05.457745 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crzrz\" (UniqueName: \"kubernetes.io/projected/99740d03-3566-4c91-95e9-c78b6cab77b6-kube-api-access-crzrz\") pod \"99740d03-3566-4c91-95e9-c78b6cab77b6\" (UID: \"99740d03-3566-4c91-95e9-c78b6cab77b6\") " Jul 25 01:18:05.460024 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:05.459995 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99740d03-3566-4c91-95e9-c78b6cab77b6-kube-api-access-crzrz" (OuterVolumeSpecName: "kube-api-access-crzrz") pod "99740d03-3566-4c91-95e9-c78b6cab77b6" (UID: "99740d03-3566-4c91-95e9-c78b6cab77b6"). InnerVolumeSpecName "kube-api-access-crzrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:18:05.558452 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:05.558424 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crzrz\" (UniqueName: \"kubernetes.io/projected/99740d03-3566-4c91-95e9-c78b6cab77b6-kube-api-access-crzrz\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:18:06.086274 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:06.086237 2554 generic.go:356] "Generic (PLEG): container finished" podID="99740d03-3566-4c91-95e9-c78b6cab77b6" containerID="e9e6b0b7e49dd43f1618324e9d2865cca3d946d6a3adb21931372918daefb748" exitCode=0 Jul 25 01:18:06.086672 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:06.086279 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vnq2h" Jul 25 01:18:06.086672 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:06.086334 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vnq2h" event={"ID":"99740d03-3566-4c91-95e9-c78b6cab77b6","Type":"ContainerDied","Data":"e9e6b0b7e49dd43f1618324e9d2865cca3d946d6a3adb21931372918daefb748"} Jul 25 01:18:06.086672 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:06.086380 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vnq2h" event={"ID":"99740d03-3566-4c91-95e9-c78b6cab77b6","Type":"ContainerDied","Data":"8f72c419f969b5e277009fa34eca9dd4164575cf128aa833d01a4aa09f86e5d0"} Jul 25 01:18:06.086672 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:06.086408 2554 scope.go:117] "RemoveContainer" containerID="e9e6b0b7e49dd43f1618324e9d2865cca3d946d6a3adb21931372918daefb748" Jul 25 01:18:06.099757 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:06.099729 2554 scope.go:117] "RemoveContainer" containerID="e9e6b0b7e49dd43f1618324e9d2865cca3d946d6a3adb21931372918daefb748" Jul 25 01:18:06.100094 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:18:06.100071 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e6b0b7e49dd43f1618324e9d2865cca3d946d6a3adb21931372918daefb748\": container with ID starting with e9e6b0b7e49dd43f1618324e9d2865cca3d946d6a3adb21931372918daefb748 not found: ID does not exist" containerID="e9e6b0b7e49dd43f1618324e9d2865cca3d946d6a3adb21931372918daefb748" Jul 25 01:18:06.100168 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:06.100106 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e6b0b7e49dd43f1618324e9d2865cca3d946d6a3adb21931372918daefb748"} err="failed to get container status \"e9e6b0b7e49dd43f1618324e9d2865cca3d946d6a3adb21931372918daefb748\": rpc error: code = NotFound desc = could not find container \"e9e6b0b7e49dd43f1618324e9d2865cca3d946d6a3adb21931372918daefb748\": container with ID starting with e9e6b0b7e49dd43f1618324e9d2865cca3d946d6a3adb21931372918daefb748 not found: ID does not exist" Jul 25 01:18:06.111531 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:06.111503 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vnq2h"] Jul 25 01:18:06.115076 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:06.115057 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vnq2h"] Jul 25 01:18:07.380119 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:07.380085 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99740d03-3566-4c91-95e9-c78b6cab77b6" path="/var/lib/kubelet/pods/99740d03-3566-4c91-95e9-c78b6cab77b6/volumes" Jul 25 01:18:12.318532 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:12.318494 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-8zlzc" Jul 25 01:18:12.318978 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:12.318559 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-8zlzc" Jul 25 01:18:12.340297 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:12.340273 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-8zlzc" Jul 25 01:18:13.144267 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:13.144236 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-8zlzc" Jul 25 01:18:16.615370 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.615331 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx"] Jul 25 01:18:16.615775 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.615633 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99740d03-3566-4c91-95e9-c78b6cab77b6" containerName="registry-server" Jul 25 01:18:16.615775 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.615644 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="99740d03-3566-4c91-95e9-c78b6cab77b6" containerName="registry-server" Jul 25 01:18:16.615775 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.615690 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="99740d03-3566-4c91-95e9-c78b6cab77b6" containerName="registry-server" Jul 25 01:18:16.620133 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.620117 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx" Jul 25 01:18:16.622091 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.622071 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cq4b8\"" Jul 25 01:18:16.626133 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.626110 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx"] Jul 25 01:18:16.715225 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.715196 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl"] Jul 25 01:18:16.718501 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.718485 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl" Jul 25 01:18:16.725724 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.725696 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl"] Jul 25 01:18:16.741444 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.741419 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8ecd12c-2b0d-42af-918e-64240ec05df6-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx\" (UID: \"b8ecd12c-2b0d-42af-918e-64240ec05df6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx" Jul 25 01:18:16.741592 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.741473 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8ecd12c-2b0d-42af-918e-64240ec05df6-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx\" (UID: \"b8ecd12c-2b0d-42af-918e-64240ec05df6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx" Jul 25 01:18:16.741592 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.741499 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqpc5\" (UniqueName: \"kubernetes.io/projected/b8ecd12c-2b0d-42af-918e-64240ec05df6-kube-api-access-cqpc5\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx\" (UID: \"b8ecd12c-2b0d-42af-918e-64240ec05df6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx" Jul 25 01:18:16.837836 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.837791 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn"] Jul 25 01:18:16.841457 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.841441 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn" Jul 25 01:18:16.842059 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.842034 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqpc5\" (UniqueName: \"kubernetes.io/projected/b8ecd12c-2b0d-42af-918e-64240ec05df6-kube-api-access-cqpc5\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx\" (UID: \"b8ecd12c-2b0d-42af-918e-64240ec05df6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx" Jul 25 01:18:16.842138 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.842121 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvgv5\" (UniqueName: \"kubernetes.io/projected/67b965b6-6327-4850-8bae-ce9374522219-kube-api-access-gvgv5\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl\" (UID: \"67b965b6-6327-4850-8bae-ce9374522219\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl" Jul 25 01:18:16.842175 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.842158 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b965b6-6327-4850-8bae-ce9374522219-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl\" (UID: \"67b965b6-6327-4850-8bae-ce9374522219\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl" Jul 25 01:18:16.842258 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.842237 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8ecd12c-2b0d-42af-918e-64240ec05df6-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx\" (UID: \"b8ecd12c-2b0d-42af-918e-64240ec05df6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx" Jul 25 01:18:16.842364 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.842343 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b965b6-6327-4850-8bae-ce9374522219-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl\" (UID: \"67b965b6-6327-4850-8bae-ce9374522219\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl" Jul 25 01:18:16.842419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.842405 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8ecd12c-2b0d-42af-918e-64240ec05df6-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx\" (UID: \"b8ecd12c-2b0d-42af-918e-64240ec05df6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx" Jul 25 01:18:16.842567 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.842549 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8ecd12c-2b0d-42af-918e-64240ec05df6-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx\" (UID: \"b8ecd12c-2b0d-42af-918e-64240ec05df6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx" Jul 25 01:18:16.842642 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.842621 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8ecd12c-2b0d-42af-918e-64240ec05df6-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx\" (UID: \"b8ecd12c-2b0d-42af-918e-64240ec05df6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx" Jul 25 01:18:16.848608 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.848385 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn"] Jul 25 01:18:16.850864 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.850841 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqpc5\" (UniqueName: \"kubernetes.io/projected/b8ecd12c-2b0d-42af-918e-64240ec05df6-kube-api-access-cqpc5\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx\" (UID: \"b8ecd12c-2b0d-42af-918e-64240ec05df6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx" Jul 25 01:18:16.930168 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.930143 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx" Jul 25 01:18:16.943554 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.943518 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b965b6-6327-4850-8bae-ce9374522219-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl\" (UID: \"67b965b6-6327-4850-8bae-ce9374522219\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl" Jul 25 01:18:16.943709 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.943597 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8299\" (UniqueName: \"kubernetes.io/projected/a274c149-ae45-4bb1-9680-779b5fa4f369-kube-api-access-h8299\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn\" (UID: \"a274c149-ae45-4bb1-9680-779b5fa4f369\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn" Jul 25 01:18:16.943709 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.943664 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a274c149-ae45-4bb1-9680-779b5fa4f369-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn\" (UID: \"a274c149-ae45-4bb1-9680-779b5fa4f369\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn" Jul 25 01:18:16.943853 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.943718 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvgv5\" (UniqueName: \"kubernetes.io/projected/67b965b6-6327-4850-8bae-ce9374522219-kube-api-access-gvgv5\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl\" (UID: \"67b965b6-6327-4850-8bae-ce9374522219\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl" Jul 25 01:18:16.944353 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.943974 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b965b6-6327-4850-8bae-ce9374522219-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl\" (UID: \"67b965b6-6327-4850-8bae-ce9374522219\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl" Jul 25 01:18:16.944353 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.943999 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b965b6-6327-4850-8bae-ce9374522219-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl\" (UID: \"67b965b6-6327-4850-8bae-ce9374522219\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl" Jul 25 01:18:16.944353 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.944072 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a274c149-ae45-4bb1-9680-779b5fa4f369-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn\" (UID: \"a274c149-ae45-4bb1-9680-779b5fa4f369\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn" Jul 25 01:18:16.944353 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.944311 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b965b6-6327-4850-8bae-ce9374522219-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl\" (UID: \"67b965b6-6327-4850-8bae-ce9374522219\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl" Jul 25 01:18:16.952037 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:16.952013 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvgv5\" (UniqueName: \"kubernetes.io/projected/67b965b6-6327-4850-8bae-ce9374522219-kube-api-access-gvgv5\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl\" (UID: \"67b965b6-6327-4850-8bae-ce9374522219\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl" Jul 25 01:18:17.029097 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.029072 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl" Jul 25 01:18:17.040403 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.040374 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm"] Jul 25 01:18:17.044739 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.044709 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a274c149-ae45-4bb1-9680-779b5fa4f369-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn\" (UID: \"a274c149-ae45-4bb1-9680-779b5fa4f369\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn" Jul 25 01:18:17.044932 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.044856 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8299\" (UniqueName: \"kubernetes.io/projected/a274c149-ae45-4bb1-9680-779b5fa4f369-kube-api-access-h8299\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn\" (UID: \"a274c149-ae45-4bb1-9680-779b5fa4f369\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn" Jul 25 01:18:17.044932 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.044928 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a274c149-ae45-4bb1-9680-779b5fa4f369-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn\" (UID: \"a274c149-ae45-4bb1-9680-779b5fa4f369\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn" Jul 25 01:18:17.045220 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.045195 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a274c149-ae45-4bb1-9680-779b5fa4f369-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn\" (UID: \"a274c149-ae45-4bb1-9680-779b5fa4f369\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn" Jul 25 01:18:17.045447 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.045340 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a274c149-ae45-4bb1-9680-779b5fa4f369-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn\" (UID: \"a274c149-ae45-4bb1-9680-779b5fa4f369\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn" Jul 25 01:18:17.045998 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.045981 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm" Jul 25 01:18:17.059854 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.059828 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm"] Jul 25 01:18:17.065912 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.065892 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8299\" (UniqueName: \"kubernetes.io/projected/a274c149-ae45-4bb1-9680-779b5fa4f369-kube-api-access-h8299\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn\" (UID: \"a274c149-ae45-4bb1-9680-779b5fa4f369\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn" Jul 25 01:18:17.091148 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.091126 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx"] Jul 25 01:18:17.094154 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:18:17.094125 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ecd12c_2b0d_42af_918e_64240ec05df6.slice/crio-75353338b179a6774bc4fcbfa94e010178f25debd60ad5594b40fcf29941da55 WatchSource:0}: Error finding container 75353338b179a6774bc4fcbfa94e010178f25debd60ad5594b40fcf29941da55: Status 404 returned error can't find the container with id 75353338b179a6774bc4fcbfa94e010178f25debd60ad5594b40fcf29941da55 Jul 25 01:18:17.139070 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.139037 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx" event={"ID":"b8ecd12c-2b0d-42af-918e-64240ec05df6","Type":"ContainerStarted","Data":"75353338b179a6774bc4fcbfa94e010178f25debd60ad5594b40fcf29941da55"} Jul 25 01:18:17.146337 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.146313 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce108073-e46c-4998-80c9-b3db2a19933f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm\" (UID: \"ce108073-e46c-4998-80c9-b3db2a19933f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm" Jul 25 01:18:17.146446 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.146364 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce108073-e46c-4998-80c9-b3db2a19933f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm\" (UID: \"ce108073-e46c-4998-80c9-b3db2a19933f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm" Jul 25 01:18:17.146446 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.146385 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m449c\" (UniqueName: \"kubernetes.io/projected/ce108073-e46c-4998-80c9-b3db2a19933f-kube-api-access-m449c\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm\" (UID: \"ce108073-e46c-4998-80c9-b3db2a19933f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm" Jul 25 01:18:17.152896 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.152869 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn" Jul 25 01:18:17.183879 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.183815 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl"] Jul 25 01:18:17.187177 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:18:17.187153 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b965b6_6327_4850_8bae_ce9374522219.slice/crio-ec89601f682003410014e23e403230dfecbff1e29bd73284f3d4224486785c9c WatchSource:0}: Error finding container ec89601f682003410014e23e403230dfecbff1e29bd73284f3d4224486785c9c: Status 404 returned error can't find the container with id ec89601f682003410014e23e403230dfecbff1e29bd73284f3d4224486785c9c Jul 25 01:18:17.247420 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.247394 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce108073-e46c-4998-80c9-b3db2a19933f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm\" (UID: \"ce108073-e46c-4998-80c9-b3db2a19933f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm" Jul 25 01:18:17.247575 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.247444 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce108073-e46c-4998-80c9-b3db2a19933f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm\" (UID: \"ce108073-e46c-4998-80c9-b3db2a19933f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm" Jul 25 01:18:17.247575 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.247464 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m449c\" (UniqueName: \"kubernetes.io/projected/ce108073-e46c-4998-80c9-b3db2a19933f-kube-api-access-m449c\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm\" (UID: \"ce108073-e46c-4998-80c9-b3db2a19933f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm" Jul 25 01:18:17.247854 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.247777 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce108073-e46c-4998-80c9-b3db2a19933f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm\" (UID: \"ce108073-e46c-4998-80c9-b3db2a19933f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm" Jul 25 01:18:17.248035 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.248018 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce108073-e46c-4998-80c9-b3db2a19933f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm\" (UID: \"ce108073-e46c-4998-80c9-b3db2a19933f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm" Jul 25 01:18:17.255938 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.255915 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m449c\" (UniqueName: \"kubernetes.io/projected/ce108073-e46c-4998-80c9-b3db2a19933f-kube-api-access-m449c\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm\" (UID: \"ce108073-e46c-4998-80c9-b3db2a19933f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm" Jul 25 01:18:17.316095 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.316073 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn"] Jul 25 01:18:17.317365 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:18:17.317339 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda274c149_ae45_4bb1_9680_779b5fa4f369.slice/crio-14c51a050cd2b141e6b197b72e9047d06d7aa3b3d0f3b14572eaf69fac1d8fe7 WatchSource:0}: Error finding container 14c51a050cd2b141e6b197b72e9047d06d7aa3b3d0f3b14572eaf69fac1d8fe7: Status 404 returned error can't find the container with id 14c51a050cd2b141e6b197b72e9047d06d7aa3b3d0f3b14572eaf69fac1d8fe7 Jul 25 01:18:17.360569 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.360546 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm" Jul 25 01:18:17.498725 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:17.498702 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm"] Jul 25 01:18:17.524555 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:18:17.524529 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce108073_e46c_4998_80c9_b3db2a19933f.slice/crio-89866e792aa94b5331563a70dc11b1fcfb866914ad3a0a31757435265ad5bb88 WatchSource:0}: Error finding container 89866e792aa94b5331563a70dc11b1fcfb866914ad3a0a31757435265ad5bb88: Status 404 returned error can't find the container with id 89866e792aa94b5331563a70dc11b1fcfb866914ad3a0a31757435265ad5bb88 Jul 25 01:18:18.148523 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:18.148485 2554 generic.go:356] "Generic (PLEG): container finished" podID="67b965b6-6327-4850-8bae-ce9374522219" containerID="f57cc5a27d8675ca85a34675bd799dcf62de02650ba2faa7349fd4dd7dd80cc4" exitCode=0 Jul 25 01:18:18.149125 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:18.148572 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl" event={"ID":"67b965b6-6327-4850-8bae-ce9374522219","Type":"ContainerDied","Data":"f57cc5a27d8675ca85a34675bd799dcf62de02650ba2faa7349fd4dd7dd80cc4"} Jul 25 01:18:18.149125 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:18.148629 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl" event={"ID":"67b965b6-6327-4850-8bae-ce9374522219","Type":"ContainerStarted","Data":"ec89601f682003410014e23e403230dfecbff1e29bd73284f3d4224486785c9c"} Jul 25 01:18:18.150380 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:18.150360 2554 generic.go:356] "Generic (PLEG): container finished" podID="b8ecd12c-2b0d-42af-918e-64240ec05df6" containerID="37486859b4234344d1849f1eddecc6fcab3736571ceccdf167a51b1f2e88811f" exitCode=0 Jul 25 01:18:18.150477 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:18.150449 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx" event={"ID":"b8ecd12c-2b0d-42af-918e-64240ec05df6","Type":"ContainerDied","Data":"37486859b4234344d1849f1eddecc6fcab3736571ceccdf167a51b1f2e88811f"} Jul 25 01:18:18.151947 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:18.151923 2554 generic.go:356] "Generic (PLEG): container finished" podID="ce108073-e46c-4998-80c9-b3db2a19933f" containerID="706638efcda9f4f832cbb5b82081e072540b42814549f69697aea47f1bd2c30f" exitCode=0 Jul 25 01:18:18.152044 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:18.151989 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm" event={"ID":"ce108073-e46c-4998-80c9-b3db2a19933f","Type":"ContainerDied","Data":"706638efcda9f4f832cbb5b82081e072540b42814549f69697aea47f1bd2c30f"} Jul 25 01:18:18.152044 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:18.152028 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm" event={"ID":"ce108073-e46c-4998-80c9-b3db2a19933f","Type":"ContainerStarted","Data":"89866e792aa94b5331563a70dc11b1fcfb866914ad3a0a31757435265ad5bb88"} Jul 25 01:18:18.153530 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:18.153491 2554 generic.go:356] "Generic (PLEG): container finished" podID="a274c149-ae45-4bb1-9680-779b5fa4f369" containerID="64bc3a8ff4e90ac96a40037ee1a7cdd0c34432046045d0edfaecbdfacd961150" exitCode=0 Jul 25 01:18:18.153629 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:18.153564 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn" event={"ID":"a274c149-ae45-4bb1-9680-779b5fa4f369","Type":"ContainerDied","Data":"64bc3a8ff4e90ac96a40037ee1a7cdd0c34432046045d0edfaecbdfacd961150"} Jul 25 01:18:18.153629 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:18.153624 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn" event={"ID":"a274c149-ae45-4bb1-9680-779b5fa4f369","Type":"ContainerStarted","Data":"14c51a050cd2b141e6b197b72e9047d06d7aa3b3d0f3b14572eaf69fac1d8fe7"} Jul 25 01:18:19.160383 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:19.160317 2554 generic.go:356] "Generic (PLEG): container finished" podID="a274c149-ae45-4bb1-9680-779b5fa4f369" containerID="8e1780d7fc839576e40c673b49b5edf0a66376f96272b61dcb0ab8ee4f00a97a" exitCode=0 Jul 25 01:18:19.160383 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:19.160376 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn" event={"ID":"a274c149-ae45-4bb1-9680-779b5fa4f369","Type":"ContainerDied","Data":"8e1780d7fc839576e40c673b49b5edf0a66376f96272b61dcb0ab8ee4f00a97a"} Jul 25 01:18:20.166569 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:20.166534 2554 generic.go:356] "Generic (PLEG): container finished" podID="67b965b6-6327-4850-8bae-ce9374522219" containerID="d9d12ad8abbe32e54885cc6a5d0d07e59a8d74381782f2a2a9ae1adcb94ee0a2" exitCode=0 Jul 25 01:18:20.167008 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:20.166620 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl" event={"ID":"67b965b6-6327-4850-8bae-ce9374522219","Type":"ContainerDied","Data":"d9d12ad8abbe32e54885cc6a5d0d07e59a8d74381782f2a2a9ae1adcb94ee0a2"} Jul 25 01:18:20.168510 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:20.168490 2554 generic.go:356] "Generic (PLEG): container finished" podID="b8ecd12c-2b0d-42af-918e-64240ec05df6" containerID="e0fbde27e2d94cf2053aa8382e75dec5708dafa46f006ac4a57f47333fdd585e" exitCode=0 Jul 25 01:18:20.168619 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:20.168574 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx" event={"ID":"b8ecd12c-2b0d-42af-918e-64240ec05df6","Type":"ContainerDied","Data":"e0fbde27e2d94cf2053aa8382e75dec5708dafa46f006ac4a57f47333fdd585e"} Jul 25 01:18:20.172234 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:20.172205 2554 generic.go:356] "Generic (PLEG): container finished" podID="ce108073-e46c-4998-80c9-b3db2a19933f" containerID="8c09c91f763a50e03b941af4ebf81320a478468af3df737130b650dfe35193de" exitCode=0 Jul 25 01:18:20.172332 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:20.172302 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm" event={"ID":"ce108073-e46c-4998-80c9-b3db2a19933f","Type":"ContainerDied","Data":"8c09c91f763a50e03b941af4ebf81320a478468af3df737130b650dfe35193de"} Jul 25 01:18:20.178829 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:20.178785 2554 generic.go:356] "Generic (PLEG): container finished" podID="a274c149-ae45-4bb1-9680-779b5fa4f369" containerID="7306cb5e60fa19604bb923a3552b1ea1f88c3515a0031442fdc8c5a7228a4504" exitCode=0 Jul 25 01:18:20.178923 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:20.178827 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn" event={"ID":"a274c149-ae45-4bb1-9680-779b5fa4f369","Type":"ContainerDied","Data":"7306cb5e60fa19604bb923a3552b1ea1f88c3515a0031442fdc8c5a7228a4504"} Jul 25 01:18:21.185584 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:21.185554 2554 generic.go:356] "Generic (PLEG): container finished" podID="67b965b6-6327-4850-8bae-ce9374522219" containerID="b796176e7189d3f57808fd3579d8b9a1a3f9ff9163e461a6a9a9d0310547b7ed" exitCode=0 Jul 25 01:18:21.186045 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:21.185641 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl" event={"ID":"67b965b6-6327-4850-8bae-ce9374522219","Type":"ContainerDied","Data":"b796176e7189d3f57808fd3579d8b9a1a3f9ff9163e461a6a9a9d0310547b7ed"} Jul 25 01:18:21.187646 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:21.187622 2554 generic.go:356] "Generic (PLEG): container finished" podID="b8ecd12c-2b0d-42af-918e-64240ec05df6" containerID="12ccd05337aec9ccfb4b43897737d6a74dea8d6561eb4f8d1fa2be211ea9ac61" exitCode=0 Jul 25 01:18:21.187766 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:21.187703 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx" event={"ID":"b8ecd12c-2b0d-42af-918e-64240ec05df6","Type":"ContainerDied","Data":"12ccd05337aec9ccfb4b43897737d6a74dea8d6561eb4f8d1fa2be211ea9ac61"} Jul 25 01:18:21.189740 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:21.189715 2554 generic.go:356] "Generic (PLEG): container finished" podID="ce108073-e46c-4998-80c9-b3db2a19933f" containerID="aadeb0c44470e44fce05aeb0ee57eeca9142cc1d3f833c3bb6febba231c058d1" exitCode=0 Jul 25 01:18:21.189869 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:21.189793 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm" event={"ID":"ce108073-e46c-4998-80c9-b3db2a19933f","Type":"ContainerDied","Data":"aadeb0c44470e44fce05aeb0ee57eeca9142cc1d3f833c3bb6febba231c058d1"} Jul 25 01:18:21.317934 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:21.317912 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn" Jul 25 01:18:21.386116 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:21.386045 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a274c149-ae45-4bb1-9680-779b5fa4f369-util\") pod \"a274c149-ae45-4bb1-9680-779b5fa4f369\" (UID: \"a274c149-ae45-4bb1-9680-779b5fa4f369\") " Jul 25 01:18:21.386253 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:21.386127 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8299\" (UniqueName: \"kubernetes.io/projected/a274c149-ae45-4bb1-9680-779b5fa4f369-kube-api-access-h8299\") pod \"a274c149-ae45-4bb1-9680-779b5fa4f369\" (UID: \"a274c149-ae45-4bb1-9680-779b5fa4f369\") " Jul 25 01:18:21.386253 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:21.386157 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a274c149-ae45-4bb1-9680-779b5fa4f369-bundle\") pod \"a274c149-ae45-4bb1-9680-779b5fa4f369\" (UID: \"a274c149-ae45-4bb1-9680-779b5fa4f369\") " Jul 25 01:18:21.386683 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:21.386655 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a274c149-ae45-4bb1-9680-779b5fa4f369-bundle" (OuterVolumeSpecName: "bundle") pod "a274c149-ae45-4bb1-9680-779b5fa4f369" (UID: "a274c149-ae45-4bb1-9680-779b5fa4f369"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:18:21.388280 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:21.388257 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a274c149-ae45-4bb1-9680-779b5fa4f369-kube-api-access-h8299" (OuterVolumeSpecName: "kube-api-access-h8299") pod "a274c149-ae45-4bb1-9680-779b5fa4f369" (UID: "a274c149-ae45-4bb1-9680-779b5fa4f369"). InnerVolumeSpecName "kube-api-access-h8299". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:18:21.391613 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:21.391590 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a274c149-ae45-4bb1-9680-779b5fa4f369-util" (OuterVolumeSpecName: "util") pod "a274c149-ae45-4bb1-9680-779b5fa4f369" (UID: "a274c149-ae45-4bb1-9680-779b5fa4f369"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:18:21.487478 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:21.487449 2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a274c149-ae45-4bb1-9680-779b5fa4f369-util\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:18:21.487478 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:21.487476 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8299\" (UniqueName: \"kubernetes.io/projected/a274c149-ae45-4bb1-9680-779b5fa4f369-kube-api-access-h8299\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:18:21.487654 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:21.487489 2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a274c149-ae45-4bb1-9680-779b5fa4f369-bundle\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:18:22.196908 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.196870 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn" event={"ID":"a274c149-ae45-4bb1-9680-779b5fa4f369","Type":"ContainerDied","Data":"14c51a050cd2b141e6b197b72e9047d06d7aa3b3d0f3b14572eaf69fac1d8fe7"} Jul 25 01:18:22.197281 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.196930 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14c51a050cd2b141e6b197b72e9047d06d7aa3b3d0f3b14572eaf69fac1d8fe7" Jul 25 01:18:22.197281 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.196888 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn" Jul 25 01:18:22.336446 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.336411 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx" Jul 25 01:18:22.369450 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.369429 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl" Jul 25 01:18:22.383989 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.383966 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm" Jul 25 01:18:22.496603 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.496520 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce108073-e46c-4998-80c9-b3db2a19933f-util\") pod \"ce108073-e46c-4998-80c9-b3db2a19933f\" (UID: \"ce108073-e46c-4998-80c9-b3db2a19933f\") " Jul 25 01:18:22.496603 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.496566 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b965b6-6327-4850-8bae-ce9374522219-bundle\") pod \"67b965b6-6327-4850-8bae-ce9374522219\" (UID: \"67b965b6-6327-4850-8bae-ce9374522219\") " Jul 25 01:18:22.496603 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.496597 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvgv5\" (UniqueName: \"kubernetes.io/projected/67b965b6-6327-4850-8bae-ce9374522219-kube-api-access-gvgv5\") pod \"67b965b6-6327-4850-8bae-ce9374522219\" (UID: \"67b965b6-6327-4850-8bae-ce9374522219\") " Jul 25 01:18:22.496916 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.496626 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8ecd12c-2b0d-42af-918e-64240ec05df6-util\") pod \"b8ecd12c-2b0d-42af-918e-64240ec05df6\" (UID: \"b8ecd12c-2b0d-42af-918e-64240ec05df6\") " Jul 25 01:18:22.496916 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.496657 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8ecd12c-2b0d-42af-918e-64240ec05df6-bundle\") pod \"b8ecd12c-2b0d-42af-918e-64240ec05df6\" (UID: \"b8ecd12c-2b0d-42af-918e-64240ec05df6\") " Jul 25 01:18:22.496916 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.496683 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce108073-e46c-4998-80c9-b3db2a19933f-bundle\") pod \"ce108073-e46c-4998-80c9-b3db2a19933f\" (UID: \"ce108073-e46c-4998-80c9-b3db2a19933f\") " Jul 25 01:18:22.496916 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.496722 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b965b6-6327-4850-8bae-ce9374522219-util\") pod \"67b965b6-6327-4850-8bae-ce9374522219\" (UID: \"67b965b6-6327-4850-8bae-ce9374522219\") " Jul 25 01:18:22.496916 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.496779 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqpc5\" (UniqueName: \"kubernetes.io/projected/b8ecd12c-2b0d-42af-918e-64240ec05df6-kube-api-access-cqpc5\") pod \"b8ecd12c-2b0d-42af-918e-64240ec05df6\" (UID: \"b8ecd12c-2b0d-42af-918e-64240ec05df6\") " Jul 25 01:18:22.496916 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.496858 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m449c\" (UniqueName: \"kubernetes.io/projected/ce108073-e46c-4998-80c9-b3db2a19933f-kube-api-access-m449c\") pod \"ce108073-e46c-4998-80c9-b3db2a19933f\" (UID: \"ce108073-e46c-4998-80c9-b3db2a19933f\") " Jul 25 01:18:22.497227 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.497188 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce108073-e46c-4998-80c9-b3db2a19933f-bundle" (OuterVolumeSpecName: "bundle") pod "ce108073-e46c-4998-80c9-b3db2a19933f" (UID: "ce108073-e46c-4998-80c9-b3db2a19933f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:18:22.497362 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.497330 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ecd12c-2b0d-42af-918e-64240ec05df6-bundle" (OuterVolumeSpecName: "bundle") pod "b8ecd12c-2b0d-42af-918e-64240ec05df6" (UID: "b8ecd12c-2b0d-42af-918e-64240ec05df6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:18:22.497362 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.497347 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b965b6-6327-4850-8bae-ce9374522219-bundle" (OuterVolumeSpecName: "bundle") pod "67b965b6-6327-4850-8bae-ce9374522219" (UID: "67b965b6-6327-4850-8bae-ce9374522219"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:18:22.497916 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.497891 2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8ecd12c-2b0d-42af-918e-64240ec05df6-bundle\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:18:22.498017 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.497918 2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce108073-e46c-4998-80c9-b3db2a19933f-bundle\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:18:22.498017 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.497934 2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b965b6-6327-4850-8bae-ce9374522219-bundle\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:18:22.499217 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.499194 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce108073-e46c-4998-80c9-b3db2a19933f-kube-api-access-m449c" (OuterVolumeSpecName: "kube-api-access-m449c") pod "ce108073-e46c-4998-80c9-b3db2a19933f" (UID: "ce108073-e46c-4998-80c9-b3db2a19933f"). InnerVolumeSpecName "kube-api-access-m449c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:18:22.499315 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.499262 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b965b6-6327-4850-8bae-ce9374522219-kube-api-access-gvgv5" (OuterVolumeSpecName: "kube-api-access-gvgv5") pod "67b965b6-6327-4850-8bae-ce9374522219" (UID: "67b965b6-6327-4850-8bae-ce9374522219"). InnerVolumeSpecName "kube-api-access-gvgv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:18:22.499626 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.499610 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ecd12c-2b0d-42af-918e-64240ec05df6-kube-api-access-cqpc5" (OuterVolumeSpecName: "kube-api-access-cqpc5") pod "b8ecd12c-2b0d-42af-918e-64240ec05df6" (UID: "b8ecd12c-2b0d-42af-918e-64240ec05df6"). InnerVolumeSpecName "kube-api-access-cqpc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:18:22.503098 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.503077 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce108073-e46c-4998-80c9-b3db2a19933f-util" (OuterVolumeSpecName: "util") pod "ce108073-e46c-4998-80c9-b3db2a19933f" (UID: "ce108073-e46c-4998-80c9-b3db2a19933f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:18:22.598294 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.598267 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m449c\" (UniqueName: \"kubernetes.io/projected/ce108073-e46c-4998-80c9-b3db2a19933f-kube-api-access-m449c\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:18:22.598294 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.598292 2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce108073-e46c-4998-80c9-b3db2a19933f-util\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:18:22.598438 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.598302 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvgv5\" (UniqueName: \"kubernetes.io/projected/67b965b6-6327-4850-8bae-ce9374522219-kube-api-access-gvgv5\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:18:22.598438 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.598311 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqpc5\" (UniqueName: \"kubernetes.io/projected/b8ecd12c-2b0d-42af-918e-64240ec05df6-kube-api-access-cqpc5\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:18:22.781439 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.781362 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b965b6-6327-4850-8bae-ce9374522219-util" (OuterVolumeSpecName: "util") pod "67b965b6-6327-4850-8bae-ce9374522219" (UID: "67b965b6-6327-4850-8bae-ce9374522219"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:18:22.795250 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.795223 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ecd12c-2b0d-42af-918e-64240ec05df6-util" (OuterVolumeSpecName: "util") pod "b8ecd12c-2b0d-42af-918e-64240ec05df6" (UID: "b8ecd12c-2b0d-42af-918e-64240ec05df6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:18:22.799994 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.799976 2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8ecd12c-2b0d-42af-918e-64240ec05df6-util\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:18:22.800062 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:22.799996 2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b965b6-6327-4850-8bae-ce9374522219-util\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:18:23.203071 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:23.203042 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx" Jul 25 01:18:23.203071 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:23.203055 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx" event={"ID":"b8ecd12c-2b0d-42af-918e-64240ec05df6","Type":"ContainerDied","Data":"75353338b179a6774bc4fcbfa94e010178f25debd60ad5594b40fcf29941da55"} Jul 25 01:18:23.203553 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:23.203140 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75353338b179a6774bc4fcbfa94e010178f25debd60ad5594b40fcf29941da55" Jul 25 01:18:23.205181 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:23.205154 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm" Jul 25 01:18:23.205307 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:23.205198 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm" event={"ID":"ce108073-e46c-4998-80c9-b3db2a19933f","Type":"ContainerDied","Data":"89866e792aa94b5331563a70dc11b1fcfb866914ad3a0a31757435265ad5bb88"} Jul 25 01:18:23.205307 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:23.205240 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89866e792aa94b5331563a70dc11b1fcfb866914ad3a0a31757435265ad5bb88" Jul 25 01:18:23.207337 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:23.207311 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl" event={"ID":"67b965b6-6327-4850-8bae-ce9374522219","Type":"ContainerDied","Data":"ec89601f682003410014e23e403230dfecbff1e29bd73284f3d4224486785c9c"} Jul 25 01:18:23.207449 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:23.207355 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec89601f682003410014e23e403230dfecbff1e29bd73284f3d4224486785c9c" Jul 25 01:18:23.207449 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:23.207326 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl" Jul 25 01:18:32.263717 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.263681 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-f8lns"] Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264014 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a274c149-ae45-4bb1-9680-779b5fa4f369" containerName="extract" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264026 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a274c149-ae45-4bb1-9680-779b5fa4f369" containerName="extract" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264039 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8ecd12c-2b0d-42af-918e-64240ec05df6" containerName="pull" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264044 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8ecd12c-2b0d-42af-918e-64240ec05df6" containerName="pull" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264049 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce108073-e46c-4998-80c9-b3db2a19933f" containerName="util" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264055 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce108073-e46c-4998-80c9-b3db2a19933f" containerName="util" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264061 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce108073-e46c-4998-80c9-b3db2a19933f" containerName="extract" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264066 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce108073-e46c-4998-80c9-b3db2a19933f" containerName="extract" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264071 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b965b6-6327-4850-8bae-ce9374522219" containerName="pull" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264076 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b965b6-6327-4850-8bae-ce9374522219" containerName="pull" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264083 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a274c149-ae45-4bb1-9680-779b5fa4f369" containerName="util" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264087 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a274c149-ae45-4bb1-9680-779b5fa4f369" containerName="util" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264093 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce108073-e46c-4998-80c9-b3db2a19933f" containerName="pull" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264097 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce108073-e46c-4998-80c9-b3db2a19933f" containerName="pull" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264103 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b965b6-6327-4850-8bae-ce9374522219" containerName="util" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264108 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b965b6-6327-4850-8bae-ce9374522219" containerName="util" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264115 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8ecd12c-2b0d-42af-918e-64240ec05df6" containerName="extract" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264120 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8ecd12c-2b0d-42af-918e-64240ec05df6" containerName="extract" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264126 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8ecd12c-2b0d-42af-918e-64240ec05df6" containerName="util" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264131 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8ecd12c-2b0d-42af-918e-64240ec05df6" containerName="util" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264136 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a274c149-ae45-4bb1-9680-779b5fa4f369" containerName="pull" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264140 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a274c149-ae45-4bb1-9680-779b5fa4f369" containerName="pull" Jul 25 01:18:32.264126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264146 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b965b6-6327-4850-8bae-ce9374522219" containerName="extract" Jul 25 01:18:32.264874 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264150 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b965b6-6327-4850-8bae-ce9374522219" containerName="extract" Jul 25 01:18:32.264874 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264196 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce108073-e46c-4998-80c9-b3db2a19933f" containerName="extract" Jul 25 01:18:32.264874 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264203 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8ecd12c-2b0d-42af-918e-64240ec05df6" containerName="extract" Jul 25 01:18:32.264874 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264209 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="67b965b6-6327-4850-8bae-ce9374522219" containerName="extract" Jul 25 01:18:32.264874 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.264213 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a274c149-ae45-4bb1-9680-779b5fa4f369" containerName="extract" Jul 25 01:18:32.269962 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.269948 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-f8lns" Jul 25 01:18:32.272198 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.272181 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-l8zrq\"" Jul 25 01:18:32.277903 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.277880 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-f8lns"] Jul 25 01:18:32.372726 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.372689 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rshtg\" (UniqueName: \"kubernetes.io/projected/6492d18e-7e0c-464f-9056-69745ec53c05-kube-api-access-rshtg\") pod \"authorino-operator-75cc8d5cb7-f8lns\" (UID: \"6492d18e-7e0c-464f-9056-69745ec53c05\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-f8lns" Jul 25 01:18:32.473714 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.473685 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshtg\" (UniqueName: \"kubernetes.io/projected/6492d18e-7e0c-464f-9056-69745ec53c05-kube-api-access-rshtg\") pod \"authorino-operator-75cc8d5cb7-f8lns\" (UID: \"6492d18e-7e0c-464f-9056-69745ec53c05\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-f8lns" Jul 25 01:18:32.484774 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.484744 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rshtg\" (UniqueName: \"kubernetes.io/projected/6492d18e-7e0c-464f-9056-69745ec53c05-kube-api-access-rshtg\") pod \"authorino-operator-75cc8d5cb7-f8lns\" (UID: \"6492d18e-7e0c-464f-9056-69745ec53c05\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-f8lns" Jul 25 01:18:32.580575 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.580511 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-f8lns" Jul 25 01:18:32.710644 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:32.710621 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-f8lns"] Jul 25 01:18:32.712665 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:18:32.712639 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6492d18e_7e0c_464f_9056_69745ec53c05.slice/crio-6d5d92f71147bcc8f9e8edfdf4951d813ff5ec4073cf834094d8664778481f0f WatchSource:0}: Error finding container 6d5d92f71147bcc8f9e8edfdf4951d813ff5ec4073cf834094d8664778481f0f: Status 404 returned error can't find the container with id 6d5d92f71147bcc8f9e8edfdf4951d813ff5ec4073cf834094d8664778481f0f Jul 25 01:18:33.256625 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:33.256592 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-f8lns" event={"ID":"6492d18e-7e0c-464f-9056-69745ec53c05","Type":"ContainerStarted","Data":"6d5d92f71147bcc8f9e8edfdf4951d813ff5ec4073cf834094d8664778481f0f"} Jul 25 01:18:35.269081 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:35.269047 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-f8lns" event={"ID":"6492d18e-7e0c-464f-9056-69745ec53c05","Type":"ContainerStarted","Data":"e8d2dc4f40fed4a86a4f41f4ea74631ef256c759987e867e482b133221c10707"} Jul 25 01:18:35.269480 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:35.269161 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-f8lns" Jul 25 01:18:35.285247 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:35.285201 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-f8lns" podStartSLOduration=1.125409506 podStartE2EDuration="3.285187527s" totalImagesPullingTime="2.159778021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:18:35.283530116 +0000 UTC m=+456.405015967" watchObservedRunningTime="2026-07-25 01:18:35.285187527 +0000 UTC m=+456.406673378" Jul 25 01:18:36.283313 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:36.283285 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv"] Jul 25 01:18:36.286432 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:36.286416 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv" Jul 25 01:18:36.288357 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:36.288336 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-xnqsc\"" Jul 25 01:18:36.297083 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:36.297061 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv"] Jul 25 01:18:36.406544 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:36.406521 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8gqz\" (UniqueName: \"kubernetes.io/projected/787c8eec-485e-4413-aef2-834d5a855287-kube-api-access-l8gqz\") pod \"limitador-operator-controller-manager-859d7b78bd-mngxv\" (UID: \"787c8eec-485e-4413-aef2-834d5a855287\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv" Jul 25 01:18:36.507337 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:36.507309 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8gqz\" (UniqueName: \"kubernetes.io/projected/787c8eec-485e-4413-aef2-834d5a855287-kube-api-access-l8gqz\") pod \"limitador-operator-controller-manager-859d7b78bd-mngxv\" (UID: \"787c8eec-485e-4413-aef2-834d5a855287\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv" Jul 25 01:18:36.518669 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:36.518647 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8gqz\" (UniqueName: \"kubernetes.io/projected/787c8eec-485e-4413-aef2-834d5a855287-kube-api-access-l8gqz\") pod \"limitador-operator-controller-manager-859d7b78bd-mngxv\" (UID: \"787c8eec-485e-4413-aef2-834d5a855287\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv" Jul 25 01:18:36.596717 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:36.596646 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv" Jul 25 01:18:36.725657 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:36.725634 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv"] Jul 25 01:18:36.728132 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:18:36.728096 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787c8eec_485e_4413_aef2_834d5a855287.slice/crio-51b0453b7ff184506ce7cade80d7d3a4c10219bc3b3b44fa59eb22223ec83a80 WatchSource:0}: Error finding container 51b0453b7ff184506ce7cade80d7d3a4c10219bc3b3b44fa59eb22223ec83a80: Status 404 returned error can't find the container with id 51b0453b7ff184506ce7cade80d7d3a4c10219bc3b3b44fa59eb22223ec83a80 Jul 25 01:18:37.280043 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:37.280009 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv" event={"ID":"787c8eec-485e-4413-aef2-834d5a855287","Type":"ContainerStarted","Data":"51b0453b7ff184506ce7cade80d7d3a4c10219bc3b3b44fa59eb22223ec83a80"} Jul 25 01:18:39.291981 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:39.291948 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv" event={"ID":"787c8eec-485e-4413-aef2-834d5a855287","Type":"ContainerStarted","Data":"89650218735a6e819ea0e818679d83c3544b382fa06a3584dba024c6f3c7b29b"} Jul 25 01:18:39.292347 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:39.292026 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv" Jul 25 01:18:39.307792 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:39.307748 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv" podStartSLOduration=1.651255411 podStartE2EDuration="3.307734083s" totalImagesPullingTime="1.656478672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:18:39.306302216 +0000 UTC m=+460.427788066" watchObservedRunningTime="2026-07-25 01:18:39.307734083 +0000 UTC m=+460.429219932" Jul 25 01:18:40.481584 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:40.481553 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-sgg9x"] Jul 25 01:18:40.484842 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:40.484825 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sgg9x" Jul 25 01:18:40.486822 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:40.486785 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 25 01:18:40.486951 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:40.486850 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-v5tvk\"" Jul 25 01:18:40.496858 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:40.496836 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-sgg9x"] Jul 25 01:18:40.640505 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:40.640469 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhqrj\" (UniqueName: \"kubernetes.io/projected/a3be6d96-aa5d-42d7-99de-1dccfafa0f88-kube-api-access-bhqrj\") pod \"dns-operator-controller-manager-864dd78c4c-sgg9x\" (UID: \"a3be6d96-aa5d-42d7-99de-1dccfafa0f88\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sgg9x" Jul 25 01:18:40.741499 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:40.741429 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhqrj\" (UniqueName: \"kubernetes.io/projected/a3be6d96-aa5d-42d7-99de-1dccfafa0f88-kube-api-access-bhqrj\") pod \"dns-operator-controller-manager-864dd78c4c-sgg9x\" (UID: \"a3be6d96-aa5d-42d7-99de-1dccfafa0f88\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sgg9x" Jul 25 01:18:40.748905 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:40.748876 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhqrj\" (UniqueName: \"kubernetes.io/projected/a3be6d96-aa5d-42d7-99de-1dccfafa0f88-kube-api-access-bhqrj\") pod \"dns-operator-controller-manager-864dd78c4c-sgg9x\" (UID: \"a3be6d96-aa5d-42d7-99de-1dccfafa0f88\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sgg9x" Jul 25 01:18:40.794979 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:40.794957 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sgg9x" Jul 25 01:18:40.925894 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:40.925860 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-sgg9x"] Jul 25 01:18:40.927174 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:18:40.927139 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3be6d96_aa5d_42d7_99de_1dccfafa0f88.slice/crio-2f9e0d0532fe9e4a55fc2c7b25a748531a462db3cc866e23b42de5a99e9c8e7f WatchSource:0}: Error finding container 2f9e0d0532fe9e4a55fc2c7b25a748531a462db3cc866e23b42de5a99e9c8e7f: Status 404 returned error can't find the container with id 2f9e0d0532fe9e4a55fc2c7b25a748531a462db3cc866e23b42de5a99e9c8e7f Jul 25 01:18:41.309012 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:41.308974 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sgg9x" event={"ID":"a3be6d96-aa5d-42d7-99de-1dccfafa0f88","Type":"ContainerStarted","Data":"2f9e0d0532fe9e4a55fc2c7b25a748531a462db3cc866e23b42de5a99e9c8e7f"} Jul 25 01:18:43.320469 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:43.320435 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sgg9x" event={"ID":"a3be6d96-aa5d-42d7-99de-1dccfafa0f88","Type":"ContainerStarted","Data":"1debf331da040a000da752df520d00737d48b50727fad8193377844e63f90d39"} Jul 25 01:18:43.320868 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:43.320565 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sgg9x" Jul 25 01:18:43.340837 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:43.340771 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sgg9x" podStartSLOduration=1.512951915 podStartE2EDuration="3.340756179s" totalImagesPullingTime="1.827804264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:18:43.338882542 +0000 UTC m=+464.460368396" watchObservedRunningTime="2026-07-25 01:18:43.340756179 +0000 UTC m=+464.462242029" Jul 25 01:18:46.276784 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:46.276751 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-f8lns" Jul 25 01:18:46.408342 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:46.408310 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk"] Jul 25 01:18:46.411592 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:46.411576 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:18:46.413679 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:46.413657 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lxtfb\"" Jul 25 01:18:46.422855 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:46.422834 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk"] Jul 25 01:18:46.590470 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:46.590380 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68d7c38c-5f66-4211-abe3-8003c0ea0623-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4vnfk\" (UID: \"68d7c38c-5f66-4211-abe3-8003c0ea0623\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:18:46.590470 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:46.590468 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpv94\" (UniqueName: \"kubernetes.io/projected/68d7c38c-5f66-4211-abe3-8003c0ea0623-kube-api-access-dpv94\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4vnfk\" (UID: \"68d7c38c-5f66-4211-abe3-8003c0ea0623\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:18:46.691842 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:46.691785 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpv94\" (UniqueName: \"kubernetes.io/projected/68d7c38c-5f66-4211-abe3-8003c0ea0623-kube-api-access-dpv94\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4vnfk\" (UID: \"68d7c38c-5f66-4211-abe3-8003c0ea0623\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:18:46.691997 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:46.691880 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68d7c38c-5f66-4211-abe3-8003c0ea0623-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4vnfk\" (UID: \"68d7c38c-5f66-4211-abe3-8003c0ea0623\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:18:46.692258 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:46.692238 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68d7c38c-5f66-4211-abe3-8003c0ea0623-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4vnfk\" (UID: \"68d7c38c-5f66-4211-abe3-8003c0ea0623\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:18:46.700262 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:46.700244 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpv94\" (UniqueName: \"kubernetes.io/projected/68d7c38c-5f66-4211-abe3-8003c0ea0623-kube-api-access-dpv94\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4vnfk\" (UID: \"68d7c38c-5f66-4211-abe3-8003c0ea0623\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:18:46.722233 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:46.722204 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:18:46.857491 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:46.857468 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk"] Jul 25 01:18:46.859916 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:18:46.859890 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d7c38c_5f66_4211_abe3_8003c0ea0623.slice/crio-0d4934bbc06b1413f620549e6622c13b6932b93d3af52032baf2174eb86c1ad3 WatchSource:0}: Error finding container 0d4934bbc06b1413f620549e6622c13b6932b93d3af52032baf2174eb86c1ad3: Status 404 returned error can't find the container with id 0d4934bbc06b1413f620549e6622c13b6932b93d3af52032baf2174eb86c1ad3 Jul 25 01:18:47.340908 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:47.340873 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" event={"ID":"68d7c38c-5f66-4211-abe3-8003c0ea0623","Type":"ContainerStarted","Data":"0d4934bbc06b1413f620549e6622c13b6932b93d3af52032baf2174eb86c1ad3"} Jul 25 01:18:50.299599 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:50.299563 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv" Jul 25 01:18:53.380905 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:53.380818 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" event={"ID":"68d7c38c-5f66-4211-abe3-8003c0ea0623","Type":"ContainerStarted","Data":"91884193600e13a0df62d17d7d06449a46d70e85bc524d862024c448fe26ad8e"} Jul 25 01:18:53.380905 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:53.380890 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:18:53.396589 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:53.396534 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" podStartSLOduration=1.150552461 podStartE2EDuration="7.396517907s" totalImagesPullingTime="6.245965446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:18:53.39444355 +0000 UTC m=+474.515929399" watchObservedRunningTime="2026-07-25 01:18:53.396517907 +0000 UTC m=+474.518003757" Jul 25 01:18:54.327941 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:54.327910 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sgg9x" Jul 25 01:18:55.771007 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:55.770981 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vksfp"] Jul 25 01:18:55.774433 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:55.774414 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vksfp" Jul 25 01:18:55.776513 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:55.776491 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 25 01:18:55.776641 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:55.776526 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cq4b8\"" Jul 25 01:18:55.776641 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:55.776528 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 25 01:18:55.781769 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:55.781525 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vksfp"] Jul 25 01:18:55.858469 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:55.858437 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtgwn\" (UniqueName: \"kubernetes.io/projected/5f2bc227-5db8-44f8-96f6-154b0c59051f-kube-api-access-qtgwn\") pod \"kuadrant-console-plugin-66b5774985-vksfp\" (UID: \"5f2bc227-5db8-44f8-96f6-154b0c59051f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vksfp" Jul 25 01:18:55.858636 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:55.858514 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f2bc227-5db8-44f8-96f6-154b0c59051f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vksfp\" (UID: \"5f2bc227-5db8-44f8-96f6-154b0c59051f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vksfp" Jul 25 01:18:55.858636 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:55.858547 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5f2bc227-5db8-44f8-96f6-154b0c59051f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vksfp\" (UID: \"5f2bc227-5db8-44f8-96f6-154b0c59051f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vksfp" Jul 25 01:18:55.959138 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:55.959106 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f2bc227-5db8-44f8-96f6-154b0c59051f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vksfp\" (UID: \"5f2bc227-5db8-44f8-96f6-154b0c59051f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vksfp" Jul 25 01:18:55.959138 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:55.959143 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5f2bc227-5db8-44f8-96f6-154b0c59051f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vksfp\" (UID: \"5f2bc227-5db8-44f8-96f6-154b0c59051f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vksfp" Jul 25 01:18:55.959338 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:55.959181 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtgwn\" (UniqueName: \"kubernetes.io/projected/5f2bc227-5db8-44f8-96f6-154b0c59051f-kube-api-access-qtgwn\") pod \"kuadrant-console-plugin-66b5774985-vksfp\" (UID: \"5f2bc227-5db8-44f8-96f6-154b0c59051f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vksfp" Jul 25 01:18:55.959338 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:18:55.959254 2554 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 25 01:18:55.959429 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:18:55.959338 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f2bc227-5db8-44f8-96f6-154b0c59051f-plugin-serving-cert podName:5f2bc227-5db8-44f8-96f6-154b0c59051f nodeName:}" failed. No retries permitted until 2026-07-25 01:18:56.459317054 +0000 UTC m=+477.580802883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/5f2bc227-5db8-44f8-96f6-154b0c59051f-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-vksfp" (UID: "5f2bc227-5db8-44f8-96f6-154b0c59051f") : secret "plugin-serving-cert" not found Jul 25 01:18:55.959792 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:55.959775 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5f2bc227-5db8-44f8-96f6-154b0c59051f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vksfp\" (UID: \"5f2bc227-5db8-44f8-96f6-154b0c59051f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vksfp" Jul 25 01:18:55.969074 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:55.969049 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtgwn\" (UniqueName: \"kubernetes.io/projected/5f2bc227-5db8-44f8-96f6-154b0c59051f-kube-api-access-qtgwn\") pod \"kuadrant-console-plugin-66b5774985-vksfp\" (UID: \"5f2bc227-5db8-44f8-96f6-154b0c59051f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vksfp" Jul 25 01:18:56.462104 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:56.462064 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f2bc227-5db8-44f8-96f6-154b0c59051f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vksfp\" (UID: \"5f2bc227-5db8-44f8-96f6-154b0c59051f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vksfp" Jul 25 01:18:56.464640 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:56.464606 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f2bc227-5db8-44f8-96f6-154b0c59051f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vksfp\" (UID: \"5f2bc227-5db8-44f8-96f6-154b0c59051f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vksfp" Jul 25 01:18:56.685192 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:56.685158 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vksfp" Jul 25 01:18:56.814270 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:56.814244 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vksfp"] Jul 25 01:18:56.816758 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:18:56.816734 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f2bc227_5db8_44f8_96f6_154b0c59051f.slice/crio-ed998ecd0b2ef8292ac90bfbb31b4e6be65197304d56782d979d19b492a43eb3 WatchSource:0}: Error finding container ed998ecd0b2ef8292ac90bfbb31b4e6be65197304d56782d979d19b492a43eb3: Status 404 returned error can't find the container with id ed998ecd0b2ef8292ac90bfbb31b4e6be65197304d56782d979d19b492a43eb3 Jul 25 01:18:57.398819 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:18:57.398768 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vksfp" event={"ID":"5f2bc227-5db8-44f8-96f6-154b0c59051f","Type":"ContainerStarted","Data":"ed998ecd0b2ef8292ac90bfbb31b4e6be65197304d56782d979d19b492a43eb3"} Jul 25 01:19:04.383334 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:04.383302 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:19:06.087930 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.087888 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk"] Jul 25 01:19:06.088374 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.088092 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" podUID="68d7c38c-5f66-4211-abe3-8003c0ea0623" containerName="manager" containerID="cri-o://91884193600e13a0df62d17d7d06449a46d70e85bc524d862024c448fe26ad8e" gracePeriod=2 Jul 25 01:19:06.091291 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.091265 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv"] Jul 25 01:19:06.097003 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:19:06.096958 2554 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"68d7c38c-5f66-4211-abe3-8003c0ea0623\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-4vnfk\": pods \"kuadrant-operator-controller-manager-8684c9ff49-4vnfk\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:19:06.104864 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.104779 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk"] Jul 25 01:19:06.105011 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.104915 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv" Jul 25 01:19:06.110845 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.110823 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh"] Jul 25 01:19:06.111846 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.111325 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68d7c38c-5f66-4211-abe3-8003c0ea0623" containerName="manager" Jul 25 01:19:06.111846 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.111346 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="68d7c38c-5f66-4211-abe3-8003c0ea0623" containerName="manager" Jul 25 01:19:06.111846 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.111451 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="68d7c38c-5f66-4211-abe3-8003c0ea0623" containerName="manager" Jul 25 01:19:06.115469 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.115447 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv"] Jul 25 01:19:06.115617 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.115551 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh" Jul 25 01:19:06.117093 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.117068 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv"] Jul 25 01:19:06.117317 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.117291 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv" podUID="787c8eec-485e-4413-aef2-834d5a855287" containerName="manager" containerID="cri-o://89650218735a6e819ea0e818679d83c3544b382fa06a3584dba024c6f3c7b29b" gracePeriod=2 Jul 25 01:19:06.119258 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.119233 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv"] Jul 25 01:19:06.129516 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:19:06.129483 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4vnfk\" is forbidden: User \"system:node:ip-10-0-135-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-43.ec2.internal' and this object" podUID="68d7c38c-5f66-4211-abe3-8003c0ea0623" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:19:06.132740 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.132340 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh"] Jul 25 01:19:06.135900 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.135879 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h9vjm"] Jul 25 01:19:06.136323 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.136304 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="787c8eec-485e-4413-aef2-834d5a855287" containerName="manager" Jul 25 01:19:06.136416 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.136325 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="787c8eec-485e-4413-aef2-834d5a855287" containerName="manager" Jul 25 01:19:06.136484 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.136435 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="787c8eec-485e-4413-aef2-834d5a855287" containerName="manager" Jul 25 01:19:06.141141 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.141124 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h9vjm" Jul 25 01:19:06.143432 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.143412 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvtg9\" (UniqueName: \"kubernetes.io/projected/4758ec31-6edd-4b01-98f6-49363191bdf5-kube-api-access-tvtg9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n9rmh\" (UID: \"4758ec31-6edd-4b01-98f6-49363191bdf5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh" Jul 25 01:19:06.143530 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.143459 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cmrt\" (UniqueName: \"kubernetes.io/projected/326aff75-54a0-4991-bf12-f2bcd3325e3c-kube-api-access-9cmrt\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bn5bv\" (UID: \"326aff75-54a0-4991-bf12-f2bcd3325e3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv" Jul 25 01:19:06.143603 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.143534 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4758ec31-6edd-4b01-98f6-49363191bdf5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n9rmh\" (UID: \"4758ec31-6edd-4b01-98f6-49363191bdf5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh" Jul 25 01:19:06.143662 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.143621 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/326aff75-54a0-4991-bf12-f2bcd3325e3c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bn5bv\" (UID: \"326aff75-54a0-4991-bf12-f2bcd3325e3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv" Jul 25 01:19:06.150601 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.150574 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h9vjm"] Jul 25 01:19:06.151924 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:19:06.151880 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mngxv\" is forbidden: User \"system:node:ip-10-0-135-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-43.ec2.internal' and this object" podUID="787c8eec-485e-4413-aef2-834d5a855287" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv" Jul 25 01:19:06.153435 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:19:06.153412 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4vnfk\" is forbidden: User \"system:node:ip-10-0-135-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-43.ec2.internal' and this object" podUID="68d7c38c-5f66-4211-abe3-8003c0ea0623" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:19:06.172039 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:19:06.172004 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mngxv\" is forbidden: User \"system:node:ip-10-0-135-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-43.ec2.internal' and this object" podUID="787c8eec-485e-4413-aef2-834d5a855287" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv" Jul 25 01:19:06.173971 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:19:06.173934 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4vnfk\" is forbidden: User \"system:node:ip-10-0-135-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-43.ec2.internal' and this object" podUID="68d7c38c-5f66-4211-abe3-8003c0ea0623" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:19:06.244036 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.244008 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtg9\" (UniqueName: \"kubernetes.io/projected/4758ec31-6edd-4b01-98f6-49363191bdf5-kube-api-access-tvtg9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n9rmh\" (UID: \"4758ec31-6edd-4b01-98f6-49363191bdf5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh" Jul 25 01:19:06.244162 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.244046 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cmrt\" (UniqueName: \"kubernetes.io/projected/326aff75-54a0-4991-bf12-f2bcd3325e3c-kube-api-access-9cmrt\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bn5bv\" (UID: \"326aff75-54a0-4991-bf12-f2bcd3325e3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv" Jul 25 01:19:06.244162 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.244072 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsdqq\" (UniqueName: \"kubernetes.io/projected/1a21652b-6337-4a5d-8205-962ad8a81b0d-kube-api-access-fsdqq\") pod \"limitador-operator-controller-manager-859d7b78bd-h9vjm\" (UID: \"1a21652b-6337-4a5d-8205-962ad8a81b0d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h9vjm" Jul 25 01:19:06.244162 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.244100 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4758ec31-6edd-4b01-98f6-49363191bdf5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n9rmh\" (UID: \"4758ec31-6edd-4b01-98f6-49363191bdf5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh" Jul 25 01:19:06.244392 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.244185 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/326aff75-54a0-4991-bf12-f2bcd3325e3c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bn5bv\" (UID: \"326aff75-54a0-4991-bf12-f2bcd3325e3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv" Jul 25 01:19:06.244451 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.244419 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4758ec31-6edd-4b01-98f6-49363191bdf5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n9rmh\" (UID: \"4758ec31-6edd-4b01-98f6-49363191bdf5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh" Jul 25 01:19:06.244593 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.244574 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/326aff75-54a0-4991-bf12-f2bcd3325e3c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bn5bv\" (UID: \"326aff75-54a0-4991-bf12-f2bcd3325e3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv" Jul 25 01:19:06.252496 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.252471 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cmrt\" (UniqueName: \"kubernetes.io/projected/326aff75-54a0-4991-bf12-f2bcd3325e3c-kube-api-access-9cmrt\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bn5bv\" (UID: \"326aff75-54a0-4991-bf12-f2bcd3325e3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv" Jul 25 01:19:06.252605 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.252543 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvtg9\" (UniqueName: \"kubernetes.io/projected/4758ec31-6edd-4b01-98f6-49363191bdf5-kube-api-access-tvtg9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n9rmh\" (UID: \"4758ec31-6edd-4b01-98f6-49363191bdf5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh" Jul 25 01:19:06.345329 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.345243 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsdqq\" (UniqueName: \"kubernetes.io/projected/1a21652b-6337-4a5d-8205-962ad8a81b0d-kube-api-access-fsdqq\") pod \"limitador-operator-controller-manager-859d7b78bd-h9vjm\" (UID: \"1a21652b-6337-4a5d-8205-962ad8a81b0d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h9vjm" Jul 25 01:19:06.353245 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.353221 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsdqq\" (UniqueName: \"kubernetes.io/projected/1a21652b-6337-4a5d-8205-962ad8a81b0d-kube-api-access-fsdqq\") pod \"limitador-operator-controller-manager-859d7b78bd-h9vjm\" (UID: \"1a21652b-6337-4a5d-8205-962ad8a81b0d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h9vjm" Jul 25 01:19:06.500486 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.500439 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv" Jul 25 01:19:06.510073 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.510043 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh" Jul 25 01:19:06.517287 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:06.517269 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h9vjm" Jul 25 01:19:18.544420 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:18.544244 2554 generic.go:356] "Generic (PLEG): container finished" podID="787c8eec-485e-4413-aef2-834d5a855287" containerID="89650218735a6e819ea0e818679d83c3544b382fa06a3584dba024c6f3c7b29b" exitCode=0 Jul 25 01:19:18.556197 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:18.554503 2554 generic.go:356] "Generic (PLEG): container finished" podID="68d7c38c-5f66-4211-abe3-8003c0ea0623" containerID="91884193600e13a0df62d17d7d06449a46d70e85bc524d862024c448fe26ad8e" exitCode=0 Jul 25 01:19:18.690509 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:18.690480 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h9vjm"] Jul 25 01:19:18.693071 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:18.693052 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv" Jul 25 01:19:18.694167 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:19:18.694113 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a21652b_6337_4a5d_8205_962ad8a81b0d.slice/crio-ef506170d5dc7062b567b335cb35e2e6fc6dd69f492d390fc9ebf9e0d1ca2995 WatchSource:0}: Error finding container ef506170d5dc7062b567b335cb35e2e6fc6dd69f492d390fc9ebf9e0d1ca2995: Status 404 returned error can't find the container with id ef506170d5dc7062b567b335cb35e2e6fc6dd69f492d390fc9ebf9e0d1ca2995 Jul 25 01:19:18.694425 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:18.694362 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh"] Jul 25 01:19:18.694701 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:19:18.694527 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4758ec31_6edd_4b01_98f6_49363191bdf5.slice/crio-89512618a7a80cd3d9435b49a3484b03e9f8bfddc7b1cac54ffdf6680d508f93 WatchSource:0}: Error finding container 89512618a7a80cd3d9435b49a3484b03e9f8bfddc7b1cac54ffdf6680d508f93: Status 404 returned error can't find the container with id 89512618a7a80cd3d9435b49a3484b03e9f8bfddc7b1cac54ffdf6680d508f93 Jul 25 01:19:18.695053 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:19:18.695020 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mngxv\" is forbidden: User \"system:node:ip-10-0-135-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-43.ec2.internal' and this object" podUID="787c8eec-485e-4413-aef2-834d5a855287" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv" Jul 25 01:19:18.699491 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:18.699470 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:19:18.701061 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:19:18.700982 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mngxv\" is forbidden: User \"system:node:ip-10-0-135-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-43.ec2.internal' and this object" podUID="787c8eec-485e-4413-aef2-834d5a855287" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv" Jul 25 01:19:18.702264 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:19:18.702235 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4vnfk\" is forbidden: User \"system:node:ip-10-0-135-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-43.ec2.internal' and this object" podUID="68d7c38c-5f66-4211-abe3-8003c0ea0623" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:19:18.719303 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:18.719229 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv"] Jul 25 01:19:18.721843 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:19:18.721819 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326aff75_54a0_4991_bf12_f2bcd3325e3c.slice/crio-81dfdd24693e9b8db66fc20435d8596ab233ba94401d80f0eb258f2861e121ac WatchSource:0}: Error finding container 81dfdd24693e9b8db66fc20435d8596ab233ba94401d80f0eb258f2861e121ac: Status 404 returned error can't find the container with id 81dfdd24693e9b8db66fc20435d8596ab233ba94401d80f0eb258f2861e121ac Jul 25 01:19:18.754994 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:18.754971 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8gqz\" (UniqueName: \"kubernetes.io/projected/787c8eec-485e-4413-aef2-834d5a855287-kube-api-access-l8gqz\") pod \"787c8eec-485e-4413-aef2-834d5a855287\" (UID: \"787c8eec-485e-4413-aef2-834d5a855287\") " Jul 25 01:19:18.755090 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:18.755022 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpv94\" (UniqueName: \"kubernetes.io/projected/68d7c38c-5f66-4211-abe3-8003c0ea0623-kube-api-access-dpv94\") pod \"68d7c38c-5f66-4211-abe3-8003c0ea0623\" (UID: \"68d7c38c-5f66-4211-abe3-8003c0ea0623\") " Jul 25 01:19:18.755090 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:18.755053 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68d7c38c-5f66-4211-abe3-8003c0ea0623-extensions-socket-volume\") pod \"68d7c38c-5f66-4211-abe3-8003c0ea0623\" (UID: \"68d7c38c-5f66-4211-abe3-8003c0ea0623\") " Jul 25 01:19:18.755585 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:18.755539 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d7c38c-5f66-4211-abe3-8003c0ea0623-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "68d7c38c-5f66-4211-abe3-8003c0ea0623" (UID: "68d7c38c-5f66-4211-abe3-8003c0ea0623"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:19:18.757474 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:18.757428 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d7c38c-5f66-4211-abe3-8003c0ea0623-kube-api-access-dpv94" (OuterVolumeSpecName: "kube-api-access-dpv94") pod "68d7c38c-5f66-4211-abe3-8003c0ea0623" (UID: "68d7c38c-5f66-4211-abe3-8003c0ea0623"). InnerVolumeSpecName "kube-api-access-dpv94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:19:18.757566 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:18.757543 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/787c8eec-485e-4413-aef2-834d5a855287-kube-api-access-l8gqz" (OuterVolumeSpecName: "kube-api-access-l8gqz") pod "787c8eec-485e-4413-aef2-834d5a855287" (UID: "787c8eec-485e-4413-aef2-834d5a855287"). InnerVolumeSpecName "kube-api-access-l8gqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:19:18.855640 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:18.855614 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8gqz\" (UniqueName: \"kubernetes.io/projected/787c8eec-485e-4413-aef2-834d5a855287-kube-api-access-l8gqz\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:19:18.855640 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:18.855639 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpv94\" (UniqueName: \"kubernetes.io/projected/68d7c38c-5f66-4211-abe3-8003c0ea0623-kube-api-access-dpv94\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:19:18.855870 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:18.855648 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68d7c38c-5f66-4211-abe3-8003c0ea0623-extensions-socket-volume\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:19:19.379408 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:19:19.379376 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4vnfk\" is forbidden: User \"system:node:ip-10-0-135-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-43.ec2.internal' and this object" podUID="68d7c38c-5f66-4211-abe3-8003c0ea0623" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:19:19.380709 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:19:19.380685 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mngxv\" is forbidden: User \"system:node:ip-10-0-135-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-43.ec2.internal' and this object" podUID="787c8eec-485e-4413-aef2-834d5a855287" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv" Jul 25 01:19:19.382292 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.382273 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68d7c38c-5f66-4211-abe3-8003c0ea0623" path="/var/lib/kubelet/pods/68d7c38c-5f66-4211-abe3-8003c0ea0623/volumes" Jul 25 01:19:19.382631 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.382609 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="787c8eec-485e-4413-aef2-834d5a855287" path="/var/lib/kubelet/pods/787c8eec-485e-4413-aef2-834d5a855287/volumes" Jul 25 01:19:19.561168 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.561138 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:19:19.561612 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.561163 2554 scope.go:117] "RemoveContainer" containerID="91884193600e13a0df62d17d7d06449a46d70e85bc524d862024c448fe26ad8e" Jul 25 01:19:19.563211 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.563176 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh" event={"ID":"4758ec31-6edd-4b01-98f6-49363191bdf5","Type":"ContainerStarted","Data":"3900281c85eca8162bbb7aa7e5c5ea620aa0a791745f2d73fcecc25316419fb3"} Jul 25 01:19:19.563433 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.563412 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh" event={"ID":"4758ec31-6edd-4b01-98f6-49363191bdf5","Type":"ContainerStarted","Data":"89512618a7a80cd3d9435b49a3484b03e9f8bfddc7b1cac54ffdf6680d508f93"} Jul 25 01:19:19.563628 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.563606 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh" Jul 25 01:19:19.563898 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:19:19.563874 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4vnfk\" is forbidden: User \"system:node:ip-10-0-135-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-43.ec2.internal' and this object" podUID="68d7c38c-5f66-4211-abe3-8003c0ea0623" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:19:19.564915 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.564894 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vksfp" event={"ID":"5f2bc227-5db8-44f8-96f6-154b0c59051f","Type":"ContainerStarted","Data":"d7330cc902d463c4b468e16600764787deac7355ea981b44630a48642183c85c"} Jul 25 01:19:19.565491 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:19:19.565463 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4vnfk\" is forbidden: User \"system:node:ip-10-0-135-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-43.ec2.internal' and this object" podUID="68d7c38c-5f66-4211-abe3-8003c0ea0623" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:19:19.566534 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.566513 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv" event={"ID":"326aff75-54a0-4991-bf12-f2bcd3325e3c","Type":"ContainerStarted","Data":"0d9ebc216da7df81ba3195c6a4e33298786cd62a23317ac6b3a2cb3b4a505dd8"} Jul 25 01:19:19.566632 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.566558 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv" event={"ID":"326aff75-54a0-4991-bf12-f2bcd3325e3c","Type":"ContainerStarted","Data":"81dfdd24693e9b8db66fc20435d8596ab233ba94401d80f0eb258f2861e121ac"} Jul 25 01:19:19.566632 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.566585 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv" Jul 25 01:19:19.567918 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.567901 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv" Jul 25 01:19:19.571126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.570778 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h9vjm" event={"ID":"1a21652b-6337-4a5d-8205-962ad8a81b0d","Type":"ContainerStarted","Data":"9f8205d2583e1c7310454629eec8d1ae87b1d7215469c7e2ff5d6541840b084f"} Jul 25 01:19:19.571126 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.570858 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h9vjm" event={"ID":"1a21652b-6337-4a5d-8205-962ad8a81b0d","Type":"ContainerStarted","Data":"ef506170d5dc7062b567b335cb35e2e6fc6dd69f492d390fc9ebf9e0d1ca2995"} Jul 25 01:19:19.571281 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.571259 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h9vjm" Jul 25 01:19:19.571833 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.571794 2554 scope.go:117] "RemoveContainer" containerID="89650218735a6e819ea0e818679d83c3544b382fa06a3584dba024c6f3c7b29b" Jul 25 01:19:19.586329 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.586291 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh" podStartSLOduration=13.586278629 podStartE2EDuration="13.586278629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:19:19.585407588 +0000 UTC m=+500.706893438" watchObservedRunningTime="2026-07-25 01:19:19.586278629 +0000 UTC m=+500.707764479" Jul 25 01:19:19.599964 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.599909 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vksfp" podStartSLOduration=2.835395323 podStartE2EDuration="24.599895936s" totalImagesPullingTime="21.764500613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:19:19.599253458 +0000 UTC m=+500.720739327" watchObservedRunningTime="2026-07-25 01:19:19.599895936 +0000 UTC m=+500.721381785" Jul 25 01:19:19.616997 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.616940 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv" podStartSLOduration=13.616924291 podStartE2EDuration="13.616924291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:19:19.615218252 +0000 UTC m=+500.736704101" watchObservedRunningTime="2026-07-25 01:19:19.616924291 +0000 UTC m=+500.738410142" Jul 25 01:19:19.631117 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:19.631014 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h9vjm" podStartSLOduration=13.631000417 podStartE2EDuration="13.631000417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:19:19.629664891 +0000 UTC m=+500.751150742" watchObservedRunningTime="2026-07-25 01:19:19.631000417 +0000 UTC m=+500.752486328" Jul 25 01:19:19.631241 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:19:19.631212 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mngxv\" is forbidden: User \"system:node:ip-10-0-135-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-43.ec2.internal' and this object" podUID="787c8eec-485e-4413-aef2-834d5a855287" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mngxv" Jul 25 01:19:19.632597 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:19:19.632576 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4vnfk\" is forbidden: User \"system:node:ip-10-0-135-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-43.ec2.internal' and this object" podUID="68d7c38c-5f66-4211-abe3-8003c0ea0623" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4vnfk" Jul 25 01:19:30.579896 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:30.579869 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv" Jul 25 01:19:30.631674 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:30.631640 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh"] Jul 25 01:19:30.631935 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:30.631910 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh" podUID="4758ec31-6edd-4b01-98f6-49363191bdf5" containerName="manager" containerID="cri-o://3900281c85eca8162bbb7aa7e5c5ea620aa0a791745f2d73fcecc25316419fb3" gracePeriod=10 Jul 25 01:19:30.633920 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:30.633896 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh" Jul 25 01:19:30.860542 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:30.860515 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb"] Jul 25 01:19:30.862748 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:30.862729 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb" Jul 25 01:19:30.877131 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:30.877110 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb"] Jul 25 01:19:30.879457 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:30.879435 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh" Jul 25 01:19:30.957252 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:30.957216 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4758ec31-6edd-4b01-98f6-49363191bdf5-extensions-socket-volume\") pod \"4758ec31-6edd-4b01-98f6-49363191bdf5\" (UID: \"4758ec31-6edd-4b01-98f6-49363191bdf5\") " Jul 25 01:19:30.957252 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:30.957254 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvtg9\" (UniqueName: \"kubernetes.io/projected/4758ec31-6edd-4b01-98f6-49363191bdf5-kube-api-access-tvtg9\") pod \"4758ec31-6edd-4b01-98f6-49363191bdf5\" (UID: \"4758ec31-6edd-4b01-98f6-49363191bdf5\") " Jul 25 01:19:30.957479 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:30.957373 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0cfc251-80e6-43a5-9155-c0563bd1dd61-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2qfpb\" (UID: \"c0cfc251-80e6-43a5-9155-c0563bd1dd61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb" Jul 25 01:19:30.957532 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:30.957492 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wvdh\" (UniqueName: \"kubernetes.io/projected/c0cfc251-80e6-43a5-9155-c0563bd1dd61-kube-api-access-7wvdh\") pod \"kuadrant-operator-controller-manager-86878c769f-2qfpb\" (UID: \"c0cfc251-80e6-43a5-9155-c0563bd1dd61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb" Jul 25 01:19:30.957667 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:30.957647 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4758ec31-6edd-4b01-98f6-49363191bdf5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4758ec31-6edd-4b01-98f6-49363191bdf5" (UID: "4758ec31-6edd-4b01-98f6-49363191bdf5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:19:30.959408 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:30.959388 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4758ec31-6edd-4b01-98f6-49363191bdf5-kube-api-access-tvtg9" (OuterVolumeSpecName: "kube-api-access-tvtg9") pod "4758ec31-6edd-4b01-98f6-49363191bdf5" (UID: "4758ec31-6edd-4b01-98f6-49363191bdf5"). InnerVolumeSpecName "kube-api-access-tvtg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:19:31.057922 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.057882 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0cfc251-80e6-43a5-9155-c0563bd1dd61-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2qfpb\" (UID: \"c0cfc251-80e6-43a5-9155-c0563bd1dd61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb" Jul 25 01:19:31.058096 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.057979 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wvdh\" (UniqueName: \"kubernetes.io/projected/c0cfc251-80e6-43a5-9155-c0563bd1dd61-kube-api-access-7wvdh\") pod \"kuadrant-operator-controller-manager-86878c769f-2qfpb\" (UID: \"c0cfc251-80e6-43a5-9155-c0563bd1dd61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb" Jul 25 01:19:31.058096 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.058025 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4758ec31-6edd-4b01-98f6-49363191bdf5-extensions-socket-volume\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:19:31.058096 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.058040 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvtg9\" (UniqueName: \"kubernetes.io/projected/4758ec31-6edd-4b01-98f6-49363191bdf5-kube-api-access-tvtg9\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:19:31.058263 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.058242 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0cfc251-80e6-43a5-9155-c0563bd1dd61-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2qfpb\" (UID: \"c0cfc251-80e6-43a5-9155-c0563bd1dd61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb" Jul 25 01:19:31.068624 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.068606 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wvdh\" (UniqueName: \"kubernetes.io/projected/c0cfc251-80e6-43a5-9155-c0563bd1dd61-kube-api-access-7wvdh\") pod \"kuadrant-operator-controller-manager-86878c769f-2qfpb\" (UID: \"c0cfc251-80e6-43a5-9155-c0563bd1dd61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb" Jul 25 01:19:31.186872 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.186837 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb" Jul 25 01:19:31.321238 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.321214 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb"] Jul 25 01:19:31.323469 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:19:31.323436 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0cfc251_80e6_43a5_9155_c0563bd1dd61.slice/crio-a1cacfc500ffd0ee6e4dbbd2b0a3d70a33ff48a3684b2a1ccc4f46920bc79e72 WatchSource:0}: Error finding container a1cacfc500ffd0ee6e4dbbd2b0a3d70a33ff48a3684b2a1ccc4f46920bc79e72: Status 404 returned error can't find the container with id a1cacfc500ffd0ee6e4dbbd2b0a3d70a33ff48a3684b2a1ccc4f46920bc79e72 Jul 25 01:19:31.583695 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.583625 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h9vjm" Jul 25 01:19:31.633382 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.633349 2554 generic.go:356] "Generic (PLEG): container finished" podID="4758ec31-6edd-4b01-98f6-49363191bdf5" containerID="3900281c85eca8162bbb7aa7e5c5ea620aa0a791745f2d73fcecc25316419fb3" exitCode=0 Jul 25 01:19:31.633558 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.633409 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh" Jul 25 01:19:31.633558 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.633438 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh" event={"ID":"4758ec31-6edd-4b01-98f6-49363191bdf5","Type":"ContainerDied","Data":"3900281c85eca8162bbb7aa7e5c5ea620aa0a791745f2d73fcecc25316419fb3"} Jul 25 01:19:31.633558 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.633492 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh" event={"ID":"4758ec31-6edd-4b01-98f6-49363191bdf5","Type":"ContainerDied","Data":"89512618a7a80cd3d9435b49a3484b03e9f8bfddc7b1cac54ffdf6680d508f93"} Jul 25 01:19:31.633558 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.633524 2554 scope.go:117] "RemoveContainer" containerID="3900281c85eca8162bbb7aa7e5c5ea620aa0a791745f2d73fcecc25316419fb3" Jul 25 01:19:31.644221 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.644176 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb" event={"ID":"c0cfc251-80e6-43a5-9155-c0563bd1dd61","Type":"ContainerStarted","Data":"aeea9114e099cf7c2e01a1f231ae6f694c85b7c275839431e6f94e98a46f9028"} Jul 25 01:19:31.644366 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.644295 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb" event={"ID":"c0cfc251-80e6-43a5-9155-c0563bd1dd61","Type":"ContainerStarted","Data":"a1cacfc500ffd0ee6e4dbbd2b0a3d70a33ff48a3684b2a1ccc4f46920bc79e72"} Jul 25 01:19:31.644493 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.644469 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb" Jul 25 01:19:31.659942 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.659924 2554 scope.go:117] "RemoveContainer" containerID="3900281c85eca8162bbb7aa7e5c5ea620aa0a791745f2d73fcecc25316419fb3" Jul 25 01:19:31.660377 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:19:31.660347 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3900281c85eca8162bbb7aa7e5c5ea620aa0a791745f2d73fcecc25316419fb3\": container with ID starting with 3900281c85eca8162bbb7aa7e5c5ea620aa0a791745f2d73fcecc25316419fb3 not found: ID does not exist" containerID="3900281c85eca8162bbb7aa7e5c5ea620aa0a791745f2d73fcecc25316419fb3" Jul 25 01:19:31.660493 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.660383 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3900281c85eca8162bbb7aa7e5c5ea620aa0a791745f2d73fcecc25316419fb3"} err="failed to get container status \"3900281c85eca8162bbb7aa7e5c5ea620aa0a791745f2d73fcecc25316419fb3\": rpc error: code = NotFound desc = could not find container \"3900281c85eca8162bbb7aa7e5c5ea620aa0a791745f2d73fcecc25316419fb3\": container with ID starting with 3900281c85eca8162bbb7aa7e5c5ea620aa0a791745f2d73fcecc25316419fb3 not found: ID does not exist" Jul 25 01:19:31.662827 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.662744 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb" podStartSLOduration=1.662731817 podStartE2EDuration="1.662731817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:19:31.660755748 +0000 UTC m=+512.782241600" watchObservedRunningTime="2026-07-25 01:19:31.662731817 +0000 UTC m=+512.784217667" Jul 25 01:19:31.674949 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.674927 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh"] Jul 25 01:19:31.680404 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:31.680382 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9rmh"] Jul 25 01:19:33.381206 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:33.381177 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4758ec31-6edd-4b01-98f6-49363191bdf5" path="/var/lib/kubelet/pods/4758ec31-6edd-4b01-98f6-49363191bdf5/volumes" Jul 25 01:19:42.652780 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:42.652754 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb" Jul 25 01:19:42.701066 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:42.701037 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv"] Jul 25 01:19:42.701248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:42.701229 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv" podUID="326aff75-54a0-4991-bf12-f2bcd3325e3c" containerName="manager" containerID="cri-o://0d9ebc216da7df81ba3195c6a4e33298786cd62a23317ac6b3a2cb3b4a505dd8" gracePeriod=10 Jul 25 01:19:43.526367 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:43.526344 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv" Jul 25 01:19:43.664566 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:43.664536 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/326aff75-54a0-4991-bf12-f2bcd3325e3c-extensions-socket-volume\") pod \"326aff75-54a0-4991-bf12-f2bcd3325e3c\" (UID: \"326aff75-54a0-4991-bf12-f2bcd3325e3c\") " Jul 25 01:19:43.664937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:43.664589 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cmrt\" (UniqueName: \"kubernetes.io/projected/326aff75-54a0-4991-bf12-f2bcd3325e3c-kube-api-access-9cmrt\") pod \"326aff75-54a0-4991-bf12-f2bcd3325e3c\" (UID: \"326aff75-54a0-4991-bf12-f2bcd3325e3c\") " Jul 25 01:19:43.664937 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:43.664906 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326aff75-54a0-4991-bf12-f2bcd3325e3c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "326aff75-54a0-4991-bf12-f2bcd3325e3c" (UID: "326aff75-54a0-4991-bf12-f2bcd3325e3c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:19:43.666886 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:43.666864 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326aff75-54a0-4991-bf12-f2bcd3325e3c-kube-api-access-9cmrt" (OuterVolumeSpecName: "kube-api-access-9cmrt") pod "326aff75-54a0-4991-bf12-f2bcd3325e3c" (UID: "326aff75-54a0-4991-bf12-f2bcd3325e3c"). InnerVolumeSpecName "kube-api-access-9cmrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:19:43.706445 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:43.706415 2554 generic.go:356] "Generic (PLEG): container finished" podID="326aff75-54a0-4991-bf12-f2bcd3325e3c" containerID="0d9ebc216da7df81ba3195c6a4e33298786cd62a23317ac6b3a2cb3b4a505dd8" exitCode=0 Jul 25 01:19:43.706592 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:43.706458 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv" Jul 25 01:19:43.706592 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:43.706465 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv" event={"ID":"326aff75-54a0-4991-bf12-f2bcd3325e3c","Type":"ContainerDied","Data":"0d9ebc216da7df81ba3195c6a4e33298786cd62a23317ac6b3a2cb3b4a505dd8"} Jul 25 01:19:43.706592 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:43.706576 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv" event={"ID":"326aff75-54a0-4991-bf12-f2bcd3325e3c","Type":"ContainerDied","Data":"81dfdd24693e9b8db66fc20435d8596ab233ba94401d80f0eb258f2861e121ac"} Jul 25 01:19:43.706834 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:43.706611 2554 scope.go:117] "RemoveContainer" containerID="0d9ebc216da7df81ba3195c6a4e33298786cd62a23317ac6b3a2cb3b4a505dd8" Jul 25 01:19:43.716102 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:43.716087 2554 scope.go:117] "RemoveContainer" containerID="0d9ebc216da7df81ba3195c6a4e33298786cd62a23317ac6b3a2cb3b4a505dd8" Jul 25 01:19:43.716376 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:19:43.716355 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9ebc216da7df81ba3195c6a4e33298786cd62a23317ac6b3a2cb3b4a505dd8\": container with ID starting with 0d9ebc216da7df81ba3195c6a4e33298786cd62a23317ac6b3a2cb3b4a505dd8 not found: ID does not exist" containerID="0d9ebc216da7df81ba3195c6a4e33298786cd62a23317ac6b3a2cb3b4a505dd8" Jul 25 01:19:43.716457 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:43.716382 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9ebc216da7df81ba3195c6a4e33298786cd62a23317ac6b3a2cb3b4a505dd8"} err="failed to get container status \"0d9ebc216da7df81ba3195c6a4e33298786cd62a23317ac6b3a2cb3b4a505dd8\": rpc error: code = NotFound desc = could not find container \"0d9ebc216da7df81ba3195c6a4e33298786cd62a23317ac6b3a2cb3b4a505dd8\": container with ID starting with 0d9ebc216da7df81ba3195c6a4e33298786cd62a23317ac6b3a2cb3b4a505dd8 not found: ID does not exist" Jul 25 01:19:43.729522 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:43.729500 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv"] Jul 25 01:19:43.735147 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:43.735128 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bn5bv"] Jul 25 01:19:43.765472 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:43.765451 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/326aff75-54a0-4991-bf12-f2bcd3325e3c-extensions-socket-volume\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:19:43.765536 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:43.765476 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cmrt\" (UniqueName: \"kubernetes.io/projected/326aff75-54a0-4991-bf12-f2bcd3325e3c-kube-api-access-9cmrt\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:19:45.381907 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:19:45.381876 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="326aff75-54a0-4991-bf12-f2bcd3325e3c" path="/var/lib/kubelet/pods/326aff75-54a0-4991-bf12-f2bcd3325e3c/volumes" Jul 25 01:20:04.181716 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.181680 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bdkwm"] Jul 25 01:20:04.182101 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.182031 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4758ec31-6edd-4b01-98f6-49363191bdf5" containerName="manager" Jul 25 01:20:04.182101 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.182042 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4758ec31-6edd-4b01-98f6-49363191bdf5" containerName="manager" Jul 25 01:20:04.182101 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.182061 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="326aff75-54a0-4991-bf12-f2bcd3325e3c" containerName="manager" Jul 25 01:20:04.182101 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.182066 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="326aff75-54a0-4991-bf12-f2bcd3325e3c" containerName="manager" Jul 25 01:20:04.182224 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.182121 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="326aff75-54a0-4991-bf12-f2bcd3325e3c" containerName="manager" Jul 25 01:20:04.182224 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.182131 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4758ec31-6edd-4b01-98f6-49363191bdf5" containerName="manager" Jul 25 01:20:04.186555 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.186539 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bdkwm" Jul 25 01:20:04.188692 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.188662 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 25 01:20:04.192676 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.192649 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bdkwm"] Jul 25 01:20:04.217157 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.217135 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c34ccafa-e56e-4cf6-8095-1e401af86b98-config-file\") pod \"limitador-limitador-64c94fd64f-bdkwm\" (UID: \"c34ccafa-e56e-4cf6-8095-1e401af86b98\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bdkwm" Jul 25 01:20:04.217289 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.217163 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nfpz\" (UniqueName: \"kubernetes.io/projected/c34ccafa-e56e-4cf6-8095-1e401af86b98-kube-api-access-5nfpz\") pod \"limitador-limitador-64c94fd64f-bdkwm\" (UID: \"c34ccafa-e56e-4cf6-8095-1e401af86b98\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bdkwm" Jul 25 01:20:04.278879 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.278844 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bdkwm"] Jul 25 01:20:04.317745 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.317712 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c34ccafa-e56e-4cf6-8095-1e401af86b98-config-file\") pod \"limitador-limitador-64c94fd64f-bdkwm\" (UID: \"c34ccafa-e56e-4cf6-8095-1e401af86b98\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bdkwm" Jul 25 01:20:04.317745 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.317744 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nfpz\" (UniqueName: \"kubernetes.io/projected/c34ccafa-e56e-4cf6-8095-1e401af86b98-kube-api-access-5nfpz\") pod \"limitador-limitador-64c94fd64f-bdkwm\" (UID: \"c34ccafa-e56e-4cf6-8095-1e401af86b98\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bdkwm" Jul 25 01:20:04.318317 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.318299 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c34ccafa-e56e-4cf6-8095-1e401af86b98-config-file\") pod \"limitador-limitador-64c94fd64f-bdkwm\" (UID: \"c34ccafa-e56e-4cf6-8095-1e401af86b98\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bdkwm" Jul 25 01:20:04.324669 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.324639 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nfpz\" (UniqueName: \"kubernetes.io/projected/c34ccafa-e56e-4cf6-8095-1e401af86b98-kube-api-access-5nfpz\") pod \"limitador-limitador-64c94fd64f-bdkwm\" (UID: \"c34ccafa-e56e-4cf6-8095-1e401af86b98\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bdkwm" Jul 25 01:20:04.498843 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.498725 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bdkwm" Jul 25 01:20:04.637593 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.637547 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bdkwm"] Jul 25 01:20:04.639125 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:20:04.639090 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34ccafa_e56e_4cf6_8095_1e401af86b98.slice/crio-a4b8b3991f7cecdfdf6872531f129a5043a0e329e2b9170b3dffbd668d626ddc WatchSource:0}: Error finding container a4b8b3991f7cecdfdf6872531f129a5043a0e329e2b9170b3dffbd668d626ddc: Status 404 returned error can't find the container with id a4b8b3991f7cecdfdf6872531f129a5043a0e329e2b9170b3dffbd668d626ddc Jul 25 01:20:04.813662 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.813582 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bdkwm" event={"ID":"c34ccafa-e56e-4cf6-8095-1e401af86b98","Type":"ContainerStarted","Data":"a4b8b3991f7cecdfdf6872531f129a5043a0e329e2b9170b3dffbd668d626ddc"} Jul 25 01:20:04.981646 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.981616 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-xjpk2"] Jul 25 01:20:04.985954 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.985932 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xjpk2" Jul 25 01:20:04.988259 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.988239 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bwb92\"" Jul 25 01:20:04.992110 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:04.992090 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-xjpk2"] Jul 25 01:20:05.024046 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:05.024022 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd5pl\" (UniqueName: \"kubernetes.io/projected/e245dd11-c957-420b-92a9-eba00546e1fe-kube-api-access-jd5pl\") pod \"authorino-58cd955675-xjpk2\" (UID: \"e245dd11-c957-420b-92a9-eba00546e1fe\") " pod="kuadrant-system/authorino-58cd955675-xjpk2" Jul 25 01:20:05.125388 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:05.125318 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd5pl\" (UniqueName: \"kubernetes.io/projected/e245dd11-c957-420b-92a9-eba00546e1fe-kube-api-access-jd5pl\") pod \"authorino-58cd955675-xjpk2\" (UID: \"e245dd11-c957-420b-92a9-eba00546e1fe\") " pod="kuadrant-system/authorino-58cd955675-xjpk2" Jul 25 01:20:05.136960 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:05.136930 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd5pl\" (UniqueName: \"kubernetes.io/projected/e245dd11-c957-420b-92a9-eba00546e1fe-kube-api-access-jd5pl\") pod \"authorino-58cd955675-xjpk2\" (UID: \"e245dd11-c957-420b-92a9-eba00546e1fe\") " pod="kuadrant-system/authorino-58cd955675-xjpk2" Jul 25 01:20:05.296905 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:05.296874 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xjpk2" Jul 25 01:20:05.447964 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:05.447895 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-xjpk2"] Jul 25 01:20:05.453066 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:20:05.453018 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode245dd11_c957_420b_92a9_eba00546e1fe.slice/crio-88f1c44bb89d2f286ebcede9b8c6e65ead1659199493366505778e4cdd5a64cc WatchSource:0}: Error finding container 88f1c44bb89d2f286ebcede9b8c6e65ead1659199493366505778e4cdd5a64cc: Status 404 returned error can't find the container with id 88f1c44bb89d2f286ebcede9b8c6e65ead1659199493366505778e4cdd5a64cc Jul 25 01:20:05.821281 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:05.821242 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xjpk2" event={"ID":"e245dd11-c957-420b-92a9-eba00546e1fe","Type":"ContainerStarted","Data":"88f1c44bb89d2f286ebcede9b8c6e65ead1659199493366505778e4cdd5a64cc"} Jul 25 01:20:09.524993 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:09.524958 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-xjpk2"] Jul 25 01:20:09.847512 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:09.847477 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xjpk2" event={"ID":"e245dd11-c957-420b-92a9-eba00546e1fe","Type":"ContainerStarted","Data":"874c8e74991612b3533aef30f710be3f0980a3a6021b05957e42360d9cfa1663"} Jul 25 01:20:09.849051 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:09.849022 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bdkwm" event={"ID":"c34ccafa-e56e-4cf6-8095-1e401af86b98","Type":"ContainerStarted","Data":"84f51fca32eef11fa8913dcb4b80f3e5512369de2803c5a07ce16b1c91bf509a"} Jul 25 01:20:09.849175 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:09.849119 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-bdkwm" Jul 25 01:20:09.862105 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:09.862057 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-xjpk2" podStartSLOduration=1.913519869 podStartE2EDuration="5.862020675s" totalImagesPullingTime="3.948500806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:20:09.86001229 +0000 UTC m=+550.981498140" watchObservedRunningTime="2026-07-25 01:20:09.862020675 +0000 UTC m=+550.983506575" Jul 25 01:20:09.874621 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:09.874582 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-bdkwm" podStartSLOduration=1.162454489 podStartE2EDuration="5.874568919s" totalImagesPullingTime="4.71211443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:20:09.873417625 +0000 UTC m=+550.994903492" watchObservedRunningTime="2026-07-25 01:20:09.874568919 +0000 UTC m=+550.996054769" Jul 25 01:20:10.853845 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:10.853783 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-xjpk2" podUID="e245dd11-c957-420b-92a9-eba00546e1fe" containerName="authorino" containerID="cri-o://874c8e74991612b3533aef30f710be3f0980a3a6021b05957e42360d9cfa1663" gracePeriod=30 Jul 25 01:20:11.095179 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:11.095156 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xjpk2" Jul 25 01:20:11.180010 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:11.179986 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jd5pl\" (UniqueName: \"kubernetes.io/projected/e245dd11-c957-420b-92a9-eba00546e1fe-kube-api-access-jd5pl\") pod \"e245dd11-c957-420b-92a9-eba00546e1fe\" (UID: \"e245dd11-c957-420b-92a9-eba00546e1fe\") " Jul 25 01:20:11.182131 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:11.182103 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e245dd11-c957-420b-92a9-eba00546e1fe-kube-api-access-jd5pl" (OuterVolumeSpecName: "kube-api-access-jd5pl") pod "e245dd11-c957-420b-92a9-eba00546e1fe" (UID: "e245dd11-c957-420b-92a9-eba00546e1fe"). InnerVolumeSpecName "kube-api-access-jd5pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:20:11.280668 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:11.280641 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd5pl\" (UniqueName: \"kubernetes.io/projected/e245dd11-c957-420b-92a9-eba00546e1fe-kube-api-access-jd5pl\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:20:11.859535 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:11.859499 2554 generic.go:356] "Generic (PLEG): container finished" podID="e245dd11-c957-420b-92a9-eba00546e1fe" containerID="874c8e74991612b3533aef30f710be3f0980a3a6021b05957e42360d9cfa1663" exitCode=0 Jul 25 01:20:11.860052 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:11.859550 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xjpk2" Jul 25 01:20:11.860052 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:11.859581 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xjpk2" event={"ID":"e245dd11-c957-420b-92a9-eba00546e1fe","Type":"ContainerDied","Data":"874c8e74991612b3533aef30f710be3f0980a3a6021b05957e42360d9cfa1663"} Jul 25 01:20:11.860052 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:11.859640 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xjpk2" event={"ID":"e245dd11-c957-420b-92a9-eba00546e1fe","Type":"ContainerDied","Data":"88f1c44bb89d2f286ebcede9b8c6e65ead1659199493366505778e4cdd5a64cc"} Jul 25 01:20:11.860052 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:11.859663 2554 scope.go:117] "RemoveContainer" containerID="874c8e74991612b3533aef30f710be3f0980a3a6021b05957e42360d9cfa1663" Jul 25 01:20:11.868172 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:11.868158 2554 scope.go:117] "RemoveContainer" containerID="874c8e74991612b3533aef30f710be3f0980a3a6021b05957e42360d9cfa1663" Jul 25 01:20:11.868411 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:20:11.868390 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"874c8e74991612b3533aef30f710be3f0980a3a6021b05957e42360d9cfa1663\": container with ID starting with 874c8e74991612b3533aef30f710be3f0980a3a6021b05957e42360d9cfa1663 not found: ID does not exist" containerID="874c8e74991612b3533aef30f710be3f0980a3a6021b05957e42360d9cfa1663" Jul 25 01:20:11.868510 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:11.868416 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874c8e74991612b3533aef30f710be3f0980a3a6021b05957e42360d9cfa1663"} err="failed to get container status \"874c8e74991612b3533aef30f710be3f0980a3a6021b05957e42360d9cfa1663\": rpc error: code = NotFound desc = could not find container \"874c8e74991612b3533aef30f710be3f0980a3a6021b05957e42360d9cfa1663\": container with ID starting with 874c8e74991612b3533aef30f710be3f0980a3a6021b05957e42360d9cfa1663 not found: ID does not exist" Jul 25 01:20:11.874611 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:11.874441 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-xjpk2"] Jul 25 01:20:11.876848 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:11.876828 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-xjpk2"] Jul 25 01:20:13.381255 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:13.381219 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e245dd11-c957-420b-92a9-eba00546e1fe" path="/var/lib/kubelet/pods/e245dd11-c957-420b-92a9-eba00546e1fe/volumes" Jul 25 01:20:20.506454 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:20.506422 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bdkwm"] Jul 25 01:20:20.507124 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:20.506635 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-bdkwm" podUID="c34ccafa-e56e-4cf6-8095-1e401af86b98" containerName="limitador" containerID="cri-o://84f51fca32eef11fa8913dcb4b80f3e5512369de2803c5a07ce16b1c91bf509a" gracePeriod=30 Jul 25 01:20:20.507869 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:20.507341 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-bdkwm" Jul 25 01:20:21.062086 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:21.062064 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bdkwm" Jul 25 01:20:21.159396 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:21.159329 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c34ccafa-e56e-4cf6-8095-1e401af86b98-config-file\") pod \"c34ccafa-e56e-4cf6-8095-1e401af86b98\" (UID: \"c34ccafa-e56e-4cf6-8095-1e401af86b98\") " Jul 25 01:20:21.159522 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:21.159437 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nfpz\" (UniqueName: \"kubernetes.io/projected/c34ccafa-e56e-4cf6-8095-1e401af86b98-kube-api-access-5nfpz\") pod \"c34ccafa-e56e-4cf6-8095-1e401af86b98\" (UID: \"c34ccafa-e56e-4cf6-8095-1e401af86b98\") " Jul 25 01:20:21.159672 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:21.159650 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34ccafa-e56e-4cf6-8095-1e401af86b98-config-file" (OuterVolumeSpecName: "config-file") pod "c34ccafa-e56e-4cf6-8095-1e401af86b98" (UID: "c34ccafa-e56e-4cf6-8095-1e401af86b98"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 01:20:21.161619 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:21.161585 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34ccafa-e56e-4cf6-8095-1e401af86b98-kube-api-access-5nfpz" (OuterVolumeSpecName: "kube-api-access-5nfpz") pod "c34ccafa-e56e-4cf6-8095-1e401af86b98" (UID: "c34ccafa-e56e-4cf6-8095-1e401af86b98"). InnerVolumeSpecName "kube-api-access-5nfpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:20:21.260474 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:21.260443 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nfpz\" (UniqueName: \"kubernetes.io/projected/c34ccafa-e56e-4cf6-8095-1e401af86b98-kube-api-access-5nfpz\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:20:21.260474 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:21.260468 2554 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c34ccafa-e56e-4cf6-8095-1e401af86b98-config-file\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:20:21.910420 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:21.910381 2554 generic.go:356] "Generic (PLEG): container finished" podID="c34ccafa-e56e-4cf6-8095-1e401af86b98" containerID="84f51fca32eef11fa8913dcb4b80f3e5512369de2803c5a07ce16b1c91bf509a" exitCode=0 Jul 25 01:20:21.910420 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:21.910423 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bdkwm" Jul 25 01:20:21.911020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:21.910469 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bdkwm" event={"ID":"c34ccafa-e56e-4cf6-8095-1e401af86b98","Type":"ContainerDied","Data":"84f51fca32eef11fa8913dcb4b80f3e5512369de2803c5a07ce16b1c91bf509a"} Jul 25 01:20:21.911020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:21.910532 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bdkwm" event={"ID":"c34ccafa-e56e-4cf6-8095-1e401af86b98","Type":"ContainerDied","Data":"a4b8b3991f7cecdfdf6872531f129a5043a0e329e2b9170b3dffbd668d626ddc"} Jul 25 01:20:21.911020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:21.910564 2554 scope.go:117] "RemoveContainer" containerID="84f51fca32eef11fa8913dcb4b80f3e5512369de2803c5a07ce16b1c91bf509a" Jul 25 01:20:21.919310 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:21.919280 2554 scope.go:117] "RemoveContainer" containerID="84f51fca32eef11fa8913dcb4b80f3e5512369de2803c5a07ce16b1c91bf509a" Jul 25 01:20:21.919571 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:20:21.919546 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84f51fca32eef11fa8913dcb4b80f3e5512369de2803c5a07ce16b1c91bf509a\": container with ID starting with 84f51fca32eef11fa8913dcb4b80f3e5512369de2803c5a07ce16b1c91bf509a not found: ID does not exist" containerID="84f51fca32eef11fa8913dcb4b80f3e5512369de2803c5a07ce16b1c91bf509a" Jul 25 01:20:21.919652 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:21.919573 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f51fca32eef11fa8913dcb4b80f3e5512369de2803c5a07ce16b1c91bf509a"} err="failed to get container status \"84f51fca32eef11fa8913dcb4b80f3e5512369de2803c5a07ce16b1c91bf509a\": rpc error: code = NotFound desc = could not find container \"84f51fca32eef11fa8913dcb4b80f3e5512369de2803c5a07ce16b1c91bf509a\": container with ID starting with 84f51fca32eef11fa8913dcb4b80f3e5512369de2803c5a07ce16b1c91bf509a not found: ID does not exist" Jul 25 01:20:21.926018 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:21.926000 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bdkwm"] Jul 25 01:20:21.930523 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:21.930505 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bdkwm"] Jul 25 01:20:23.381214 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:23.381185 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c34ccafa-e56e-4cf6-8095-1e401af86b98" path="/var/lib/kubelet/pods/c34ccafa-e56e-4cf6-8095-1e401af86b98/volumes" Jul 25 01:20:26.305567 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.305526 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zhxb2"] Jul 25 01:20:26.306068 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.305910 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e245dd11-c957-420b-92a9-eba00546e1fe" containerName="authorino" Jul 25 01:20:26.306068 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.305927 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e245dd11-c957-420b-92a9-eba00546e1fe" containerName="authorino" Jul 25 01:20:26.306068 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.305944 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c34ccafa-e56e-4cf6-8095-1e401af86b98" containerName="limitador" Jul 25 01:20:26.306068 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.305952 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c34ccafa-e56e-4cf6-8095-1e401af86b98" containerName="limitador" Jul 25 01:20:26.306068 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.306036 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c34ccafa-e56e-4cf6-8095-1e401af86b98" containerName="limitador" Jul 25 01:20:26.306068 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.306049 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e245dd11-c957-420b-92a9-eba00546e1fe" containerName="authorino" Jul 25 01:20:26.310374 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.310353 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhxb2" Jul 25 01:20:26.312581 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.312551 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 25 01:20:26.312673 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.312644 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 25 01:20:26.312673 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.312652 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-rcq8h\"" Jul 25 01:20:26.313258 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.313237 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 25 01:20:26.315991 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.315970 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zhxb2"] Jul 25 01:20:26.401474 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.401437 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx9f9\" (UniqueName: \"kubernetes.io/projected/8e2dfe80-c5c1-470a-832a-87088b6123a4-kube-api-access-mx9f9\") pod \"postgres-69fb4594cf-zhxb2\" (UID: \"8e2dfe80-c5c1-470a-832a-87088b6123a4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhxb2" Jul 25 01:20:26.401639 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.401501 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8e2dfe80-c5c1-470a-832a-87088b6123a4-data\") pod \"postgres-69fb4594cf-zhxb2\" (UID: \"8e2dfe80-c5c1-470a-832a-87088b6123a4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhxb2" Jul 25 01:20:26.502082 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.502050 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8e2dfe80-c5c1-470a-832a-87088b6123a4-data\") pod \"postgres-69fb4594cf-zhxb2\" (UID: \"8e2dfe80-c5c1-470a-832a-87088b6123a4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhxb2" Jul 25 01:20:26.502248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.502137 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx9f9\" (UniqueName: \"kubernetes.io/projected/8e2dfe80-c5c1-470a-832a-87088b6123a4-kube-api-access-mx9f9\") pod \"postgres-69fb4594cf-zhxb2\" (UID: \"8e2dfe80-c5c1-470a-832a-87088b6123a4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhxb2" Jul 25 01:20:26.502476 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.502456 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8e2dfe80-c5c1-470a-832a-87088b6123a4-data\") pod \"postgres-69fb4594cf-zhxb2\" (UID: \"8e2dfe80-c5c1-470a-832a-87088b6123a4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhxb2" Jul 25 01:20:26.510425 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.510396 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx9f9\" (UniqueName: \"kubernetes.io/projected/8e2dfe80-c5c1-470a-832a-87088b6123a4-kube-api-access-mx9f9\") pod \"postgres-69fb4594cf-zhxb2\" (UID: \"8e2dfe80-c5c1-470a-832a-87088b6123a4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhxb2" Jul 25 01:20:26.622278 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.622186 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhxb2" Jul 25 01:20:26.754633 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.754606 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zhxb2"] Jul 25 01:20:26.756658 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:20:26.756628 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2dfe80_c5c1_470a_832a_87088b6123a4.slice/crio-af179e4c89bf8a384dfd6d86b9d6a6bc9870d55e40ff262bad954c51387bf997 WatchSource:0}: Error finding container af179e4c89bf8a384dfd6d86b9d6a6bc9870d55e40ff262bad954c51387bf997: Status 404 returned error can't find the container with id af179e4c89bf8a384dfd6d86b9d6a6bc9870d55e40ff262bad954c51387bf997 Jul 25 01:20:26.938610 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:26.938575 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhxb2" event={"ID":"8e2dfe80-c5c1-470a-832a-87088b6123a4","Type":"ContainerStarted","Data":"af179e4c89bf8a384dfd6d86b9d6a6bc9870d55e40ff262bad954c51387bf997"} Jul 25 01:20:31.968955 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:31.968858 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhxb2" event={"ID":"8e2dfe80-c5c1-470a-832a-87088b6123a4","Type":"ContainerStarted","Data":"ac69dcb1b087456ce1f89fa622899aff19d0e552c7a6867aaf5f7b40aa4a5cbc"} Jul 25 01:20:31.969421 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:31.968980 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhxb2" Jul 25 01:20:31.986575 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:31.986522 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhxb2" podStartSLOduration=1.073587716 podStartE2EDuration="5.986508026s" totalImagesPullingTime="4.91292031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:20:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:20:31.984655985 +0000 UTC m=+573.106141837" watchObservedRunningTime="2026-07-25 01:20:31.986508026 +0000 UTC m=+573.107993876" Jul 25 01:20:38.003211 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:20:38.003179 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zhxb2" Jul 25 01:21:21.041162 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.041087 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4"] Jul 25 01:21:21.044345 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.044329 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4" Jul 25 01:21:21.046136 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.046101 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 25 01:21:21.046136 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.046130 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5zznl\"" Jul 25 01:21:21.052941 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.052913 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4"] Jul 25 01:21:21.141577 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.141549 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceba955f-e2c5-411a-a455-b0dd171b9612-openshift-service-ca\") pod \"maas-api-5f8f8b9fbc-vmzj4\" (UID: \"ceba955f-e2c5-411a-a455-b0dd171b9612\") " pod="odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4" Jul 25 01:21:21.141734 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.141588 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceba955f-e2c5-411a-a455-b0dd171b9612-maas-api-tls\") pod \"maas-api-5f8f8b9fbc-vmzj4\" (UID: \"ceba955f-e2c5-411a-a455-b0dd171b9612\") " pod="odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4" Jul 25 01:21:21.141734 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.141613 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9rqb\" (UniqueName: \"kubernetes.io/projected/ceba955f-e2c5-411a-a455-b0dd171b9612-kube-api-access-n9rqb\") pod \"maas-api-5f8f8b9fbc-vmzj4\" (UID: \"ceba955f-e2c5-411a-a455-b0dd171b9612\") " pod="odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4" Jul 25 01:21:21.242380 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.242342 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9rqb\" (UniqueName: \"kubernetes.io/projected/ceba955f-e2c5-411a-a455-b0dd171b9612-kube-api-access-n9rqb\") pod \"maas-api-5f8f8b9fbc-vmzj4\" (UID: \"ceba955f-e2c5-411a-a455-b0dd171b9612\") " pod="odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4" Jul 25 01:21:21.242569 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.242433 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceba955f-e2c5-411a-a455-b0dd171b9612-openshift-service-ca\") pod \"maas-api-5f8f8b9fbc-vmzj4\" (UID: \"ceba955f-e2c5-411a-a455-b0dd171b9612\") " pod="odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4" Jul 25 01:21:21.242569 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.242474 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceba955f-e2c5-411a-a455-b0dd171b9612-maas-api-tls\") pod \"maas-api-5f8f8b9fbc-vmzj4\" (UID: \"ceba955f-e2c5-411a-a455-b0dd171b9612\") " pod="odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4" Jul 25 01:21:21.242697 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:21:21.242591 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 25 01:21:21.242697 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:21:21.242662 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceba955f-e2c5-411a-a455-b0dd171b9612-maas-api-tls podName:ceba955f-e2c5-411a-a455-b0dd171b9612 nodeName:}" failed. No retries permitted until 2026-07-25 01:21:21.742639182 +0000 UTC m=+622.864125017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ceba955f-e2c5-411a-a455-b0dd171b9612-maas-api-tls") pod "maas-api-5f8f8b9fbc-vmzj4" (UID: "ceba955f-e2c5-411a-a455-b0dd171b9612") : secret "maas-api-serving-cert" not found Jul 25 01:21:21.243086 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.243067 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceba955f-e2c5-411a-a455-b0dd171b9612-openshift-service-ca\") pod \"maas-api-5f8f8b9fbc-vmzj4\" (UID: \"ceba955f-e2c5-411a-a455-b0dd171b9612\") " pod="odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4" Jul 25 01:21:21.250454 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.250429 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9rqb\" (UniqueName: \"kubernetes.io/projected/ceba955f-e2c5-411a-a455-b0dd171b9612-kube-api-access-n9rqb\") pod \"maas-api-5f8f8b9fbc-vmzj4\" (UID: \"ceba955f-e2c5-411a-a455-b0dd171b9612\") " pod="odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4" Jul 25 01:21:21.657522 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.657485 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-cpblz"] Jul 25 01:21:21.660889 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.660873 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-cpblz" Jul 25 01:21:21.663881 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.663838 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-fvht6\"" Jul 25 01:21:21.663997 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.663970 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 25 01:21:21.672279 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.672253 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-cpblz"] Jul 25 01:21:21.748225 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.748193 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceba955f-e2c5-411a-a455-b0dd171b9612-maas-api-tls\") pod \"maas-api-5f8f8b9fbc-vmzj4\" (UID: \"ceba955f-e2c5-411a-a455-b0dd171b9612\") " pod="odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4" Jul 25 01:21:21.748384 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.748241 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/834d00ec-f37a-4e16-a9c1-c7c77d337fcf-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-cpblz\" (UID: \"834d00ec-f37a-4e16-a9c1-c7c77d337fcf\") " pod="openshift-ingress/payload-processing-5cc6cd444b-cpblz" Jul 25 01:21:21.748384 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.748275 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49lrj\" (UniqueName: \"kubernetes.io/projected/834d00ec-f37a-4e16-a9c1-c7c77d337fcf-kube-api-access-49lrj\") pod \"payload-processing-5cc6cd444b-cpblz\" (UID: \"834d00ec-f37a-4e16-a9c1-c7c77d337fcf\") " pod="openshift-ingress/payload-processing-5cc6cd444b-cpblz" Jul 25 01:21:21.750676 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.750648 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceba955f-e2c5-411a-a455-b0dd171b9612-maas-api-tls\") pod \"maas-api-5f8f8b9fbc-vmzj4\" (UID: \"ceba955f-e2c5-411a-a455-b0dd171b9612\") " pod="odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4" Jul 25 01:21:21.848945 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.848906 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49lrj\" (UniqueName: \"kubernetes.io/projected/834d00ec-f37a-4e16-a9c1-c7c77d337fcf-kube-api-access-49lrj\") pod \"payload-processing-5cc6cd444b-cpblz\" (UID: \"834d00ec-f37a-4e16-a9c1-c7c77d337fcf\") " pod="openshift-ingress/payload-processing-5cc6cd444b-cpblz" Jul 25 01:21:21.849123 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.848996 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/834d00ec-f37a-4e16-a9c1-c7c77d337fcf-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-cpblz\" (UID: \"834d00ec-f37a-4e16-a9c1-c7c77d337fcf\") " pod="openshift-ingress/payload-processing-5cc6cd444b-cpblz" Jul 25 01:21:21.849605 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.849587 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/834d00ec-f37a-4e16-a9c1-c7c77d337fcf-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-cpblz\" (UID: \"834d00ec-f37a-4e16-a9c1-c7c77d337fcf\") " pod="openshift-ingress/payload-processing-5cc6cd444b-cpblz" Jul 25 01:21:21.856284 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.856252 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49lrj\" (UniqueName: \"kubernetes.io/projected/834d00ec-f37a-4e16-a9c1-c7c77d337fcf-kube-api-access-49lrj\") pod \"payload-processing-5cc6cd444b-cpblz\" (UID: \"834d00ec-f37a-4e16-a9c1-c7c77d337fcf\") " pod="openshift-ingress/payload-processing-5cc6cd444b-cpblz" Jul 25 01:21:21.956072 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.956043 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4" Jul 25 01:21:21.972284 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:21.972257 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-cpblz" Jul 25 01:21:22.101710 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:21:22.101465 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceba955f_e2c5_411a_a455_b0dd171b9612.slice/crio-56bcc3c89489823f2deeb51f9238f6553345a6be6e27a62bdda64d3e27cd0014 WatchSource:0}: Error finding container 56bcc3c89489823f2deeb51f9238f6553345a6be6e27a62bdda64d3e27cd0014: Status 404 returned error can't find the container with id 56bcc3c89489823f2deeb51f9238f6553345a6be6e27a62bdda64d3e27cd0014 Jul 25 01:21:22.102078 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:22.101706 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4"] Jul 25 01:21:22.103314 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:22.103298 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 25 01:21:22.210896 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:22.210815 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4" event={"ID":"ceba955f-e2c5-411a-a455-b0dd171b9612","Type":"ContainerStarted","Data":"56bcc3c89489823f2deeb51f9238f6553345a6be6e27a62bdda64d3e27cd0014"} Jul 25 01:21:22.328879 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:22.328854 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-cpblz"] Jul 25 01:21:22.331662 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:21:22.331636 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834d00ec_f37a_4e16_a9c1_c7c77d337fcf.slice/crio-26d3c749483be5fadbfe727bc9bd3156ace23b7947ea631c53bdfef6a317884c WatchSource:0}: Error finding container 26d3c749483be5fadbfe727bc9bd3156ace23b7947ea631c53bdfef6a317884c: Status 404 returned error can't find the container with id 26d3c749483be5fadbfe727bc9bd3156ace23b7947ea631c53bdfef6a317884c Jul 25 01:21:23.223953 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:23.223918 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-cpblz" event={"ID":"834d00ec-f37a-4e16-a9c1-c7c77d337fcf","Type":"ContainerStarted","Data":"26d3c749483be5fadbfe727bc9bd3156ace23b7947ea631c53bdfef6a317884c"} Jul 25 01:21:25.235089 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:25.235000 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-cpblz" event={"ID":"834d00ec-f37a-4e16-a9c1-c7c77d337fcf","Type":"ContainerStarted","Data":"9e94ac8b5a86e984b33b17e51b1ce8b30f268d3979fed37bd7900688549e4f05"} Jul 25 01:21:25.235089 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:25.235062 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-cpblz" Jul 25 01:21:25.236558 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:25.236535 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4" event={"ID":"ceba955f-e2c5-411a-a455-b0dd171b9612","Type":"ContainerStarted","Data":"66c5aeda61141536e7f79a77d3f195d2ec74184cb355654c2bb1699765c56c7a"} Jul 25 01:21:25.236669 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:25.236654 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4" Jul 25 01:21:25.250219 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:25.250182 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-cpblz" podStartSLOduration=1.665158117 podStartE2EDuration="4.250170637s" totalImagesPullingTime="2.58501252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:21:25.248854131 +0000 UTC m=+626.370339981" watchObservedRunningTime="2026-07-25 01:21:25.250170637 +0000 UTC m=+626.371656486" Jul 25 01:21:25.263938 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:25.263895 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4" podStartSLOduration=1.451197335 podStartE2EDuration="4.263881687s" totalImagesPullingTime="2.812684352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:21:25.262977935 +0000 UTC m=+626.384463784" watchObservedRunningTime="2026-07-25 01:21:25.263881687 +0000 UTC m=+626.385367545" Jul 25 01:21:31.247642 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:31.247617 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5f8f8b9fbc-vmzj4" Jul 25 01:21:36.244465 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:36.244433 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-cpblz" Jul 25 01:21:49.017774 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.017741 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v"] Jul 25 01:21:49.025663 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.025638 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.028408 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.028377 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 25 01:21:49.028532 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.028377 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 25 01:21:49.028532 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.028377 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rj5g5\"" Jul 25 01:21:49.028532 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.028507 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 25 01:21:49.033162 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.033140 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v"] Jul 25 01:21:49.179354 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.179326 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ef0353-b8e3-4e53-bcdd-17d6750023d1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v\" (UID: \"59ef0353-b8e3-4e53-bcdd-17d6750023d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.179354 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.179358 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ef0353-b8e3-4e53-bcdd-17d6750023d1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v\" (UID: \"59ef0353-b8e3-4e53-bcdd-17d6750023d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.179584 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.179394 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ef0353-b8e3-4e53-bcdd-17d6750023d1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v\" (UID: \"59ef0353-b8e3-4e53-bcdd-17d6750023d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.179584 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.179492 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ef0353-b8e3-4e53-bcdd-17d6750023d1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v\" (UID: \"59ef0353-b8e3-4e53-bcdd-17d6750023d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.179584 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.179543 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ef0353-b8e3-4e53-bcdd-17d6750023d1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v\" (UID: \"59ef0353-b8e3-4e53-bcdd-17d6750023d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.179712 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.179607 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfqvf\" (UniqueName: \"kubernetes.io/projected/59ef0353-b8e3-4e53-bcdd-17d6750023d1-kube-api-access-gfqvf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v\" (UID: \"59ef0353-b8e3-4e53-bcdd-17d6750023d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.280547 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.280469 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ef0353-b8e3-4e53-bcdd-17d6750023d1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v\" (UID: \"59ef0353-b8e3-4e53-bcdd-17d6750023d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.280547 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.280509 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ef0353-b8e3-4e53-bcdd-17d6750023d1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v\" (UID: \"59ef0353-b8e3-4e53-bcdd-17d6750023d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.280761 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.280560 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ef0353-b8e3-4e53-bcdd-17d6750023d1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v\" (UID: \"59ef0353-b8e3-4e53-bcdd-17d6750023d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.280761 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.280614 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ef0353-b8e3-4e53-bcdd-17d6750023d1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v\" (UID: \"59ef0353-b8e3-4e53-bcdd-17d6750023d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.280761 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.280651 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ef0353-b8e3-4e53-bcdd-17d6750023d1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v\" (UID: \"59ef0353-b8e3-4e53-bcdd-17d6750023d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.280761 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.280678 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfqvf\" (UniqueName: \"kubernetes.io/projected/59ef0353-b8e3-4e53-bcdd-17d6750023d1-kube-api-access-gfqvf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v\" (UID: \"59ef0353-b8e3-4e53-bcdd-17d6750023d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.281111 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.281085 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ef0353-b8e3-4e53-bcdd-17d6750023d1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v\" (UID: \"59ef0353-b8e3-4e53-bcdd-17d6750023d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.281441 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.281282 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ef0353-b8e3-4e53-bcdd-17d6750023d1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v\" (UID: \"59ef0353-b8e3-4e53-bcdd-17d6750023d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.281552 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.281477 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ef0353-b8e3-4e53-bcdd-17d6750023d1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v\" (UID: \"59ef0353-b8e3-4e53-bcdd-17d6750023d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.283141 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.283120 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ef0353-b8e3-4e53-bcdd-17d6750023d1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v\" (UID: \"59ef0353-b8e3-4e53-bcdd-17d6750023d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.283264 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.283246 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ef0353-b8e3-4e53-bcdd-17d6750023d1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v\" (UID: \"59ef0353-b8e3-4e53-bcdd-17d6750023d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.289610 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.289585 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfqvf\" (UniqueName: \"kubernetes.io/projected/59ef0353-b8e3-4e53-bcdd-17d6750023d1-kube-api-access-gfqvf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v\" (UID: \"59ef0353-b8e3-4e53-bcdd-17d6750023d1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.337667 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.337638 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:49.470379 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:49.470352 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v"] Jul 25 01:21:49.472433 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:21:49.472405 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ef0353_b8e3_4e53_bcdd_17d6750023d1.slice/crio-94834a784d858270ddf740b0605dda2a9b5210d7eda8ccd2ea73d569573f6aae WatchSource:0}: Error finding container 94834a784d858270ddf740b0605dda2a9b5210d7eda8ccd2ea73d569573f6aae: Status 404 returned error can't find the container with id 94834a784d858270ddf740b0605dda2a9b5210d7eda8ccd2ea73d569573f6aae Jul 25 01:21:50.361711 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:50.361665 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" event={"ID":"59ef0353-b8e3-4e53-bcdd-17d6750023d1","Type":"ContainerStarted","Data":"94834a784d858270ddf740b0605dda2a9b5210d7eda8ccd2ea73d569573f6aae"} Jul 25 01:21:51.370815 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:51.370769 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" event={"ID":"59ef0353-b8e3-4e53-bcdd-17d6750023d1","Type":"ContainerStarted","Data":"36459894180c62c679ea9d6abe4573a50100b1f3fcbed1ed1c81e93dca05899d"} Jul 25 01:21:51.388251 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:51.388202 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" podStartSLOduration=2.322571854 podStartE2EDuration="3.388187696s" totalImagesPullingTime="1.065615842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:21:51.386197187 +0000 UTC m=+652.507683045" watchObservedRunningTime="2026-07-25 01:21:51.388187696 +0000 UTC m=+652.509673549" Jul 25 01:21:59.338705 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:59.338671 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:59.339091 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:59.338753 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:59.350920 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:59.350895 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:21:59.418792 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:21:59.418766 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v" Jul 25 01:22:27.423616 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.423574 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m"] Jul 25 01:22:27.439529 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.439496 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m"] Jul 25 01:22:27.439877 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.439858 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.443655 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.443607 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 25 01:22:27.497877 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.497842 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97b0365-8e1c-4c23-bf4e-b590985c6825-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m\" (UID: \"b97b0365-8e1c-4c23-bf4e-b590985c6825\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.498007 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.497898 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97b0365-8e1c-4c23-bf4e-b590985c6825-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m\" (UID: \"b97b0365-8e1c-4c23-bf4e-b590985c6825\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.498054 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.498029 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97b0365-8e1c-4c23-bf4e-b590985c6825-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m\" (UID: \"b97b0365-8e1c-4c23-bf4e-b590985c6825\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.498108 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.498056 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97b0365-8e1c-4c23-bf4e-b590985c6825-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m\" (UID: \"b97b0365-8e1c-4c23-bf4e-b590985c6825\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.498108 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.498085 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97b0365-8e1c-4c23-bf4e-b590985c6825-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m\" (UID: \"b97b0365-8e1c-4c23-bf4e-b590985c6825\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.498198 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.498120 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqfh2\" (UniqueName: \"kubernetes.io/projected/b97b0365-8e1c-4c23-bf4e-b590985c6825-kube-api-access-sqfh2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m\" (UID: \"b97b0365-8e1c-4c23-bf4e-b590985c6825\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.599213 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.599179 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqfh2\" (UniqueName: \"kubernetes.io/projected/b97b0365-8e1c-4c23-bf4e-b590985c6825-kube-api-access-sqfh2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m\" (UID: \"b97b0365-8e1c-4c23-bf4e-b590985c6825\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.599392 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.599226 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97b0365-8e1c-4c23-bf4e-b590985c6825-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m\" (UID: \"b97b0365-8e1c-4c23-bf4e-b590985c6825\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.599392 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.599262 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97b0365-8e1c-4c23-bf4e-b590985c6825-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m\" (UID: \"b97b0365-8e1c-4c23-bf4e-b590985c6825\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.599392 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.599347 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97b0365-8e1c-4c23-bf4e-b590985c6825-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m\" (UID: \"b97b0365-8e1c-4c23-bf4e-b590985c6825\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.599392 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.599371 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97b0365-8e1c-4c23-bf4e-b590985c6825-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m\" (UID: \"b97b0365-8e1c-4c23-bf4e-b590985c6825\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.599631 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.599402 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97b0365-8e1c-4c23-bf4e-b590985c6825-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m\" (UID: \"b97b0365-8e1c-4c23-bf4e-b590985c6825\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.599714 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.599691 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97b0365-8e1c-4c23-bf4e-b590985c6825-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m\" (UID: \"b97b0365-8e1c-4c23-bf4e-b590985c6825\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.599762 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.599750 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97b0365-8e1c-4c23-bf4e-b590985c6825-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m\" (UID: \"b97b0365-8e1c-4c23-bf4e-b590985c6825\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.599827 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.599758 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97b0365-8e1c-4c23-bf4e-b590985c6825-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m\" (UID: \"b97b0365-8e1c-4c23-bf4e-b590985c6825\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.601703 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.601682 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97b0365-8e1c-4c23-bf4e-b590985c6825-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m\" (UID: \"b97b0365-8e1c-4c23-bf4e-b590985c6825\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.601972 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.601955 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97b0365-8e1c-4c23-bf4e-b590985c6825-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m\" (UID: \"b97b0365-8e1c-4c23-bf4e-b590985c6825\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.606214 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.606196 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqfh2\" (UniqueName: \"kubernetes.io/projected/b97b0365-8e1c-4c23-bf4e-b590985c6825-kube-api-access-sqfh2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m\" (UID: \"b97b0365-8e1c-4c23-bf4e-b590985c6825\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.753295 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.753226 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:27.894688 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:27.894661 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m"] Jul 25 01:22:27.896790 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:22:27.896763 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97b0365_8e1c_4c23_bf4e_b590985c6825.slice/crio-0b6ccfc5779d2116bea8ba07436c2ee6c5b3d21e94c761a8ebfaeffb5f90eee1 WatchSource:0}: Error finding container 0b6ccfc5779d2116bea8ba07436c2ee6c5b3d21e94c761a8ebfaeffb5f90eee1: Status 404 returned error can't find the container with id 0b6ccfc5779d2116bea8ba07436c2ee6c5b3d21e94c761a8ebfaeffb5f90eee1 Jul 25 01:22:28.556738 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:28.556703 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" event={"ID":"b97b0365-8e1c-4c23-bf4e-b590985c6825","Type":"ContainerStarted","Data":"289a70601471ac04dfeda50c4ef8969654be269a4bc70eabb0e00f6885bb0a89"} Jul 25 01:22:28.557130 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:28.556769 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" event={"ID":"b97b0365-8e1c-4c23-bf4e-b590985c6825","Type":"ContainerStarted","Data":"0b6ccfc5779d2116bea8ba07436c2ee6c5b3d21e94c761a8ebfaeffb5f90eee1"} Jul 25 01:22:28.577359 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:28.577306 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" podStartSLOduration=1.360586429 podStartE2EDuration="1.577290474s" totalImagesPullingTime="216.704045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:22:28.575473918 +0000 UTC m=+689.696959769" watchObservedRunningTime="2026-07-25 01:22:28.577290474 +0000 UTC m=+689.698776370" Jul 25 01:22:37.754133 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:37.754099 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:37.754606 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:37.754157 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:37.766409 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:37.766376 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:38.619379 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:38.619305 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m" Jul 25 01:22:48.325267 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.325227 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6"] Jul 25 01:22:48.328996 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.328976 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.330917 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.330889 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 25 01:22:48.338563 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.338540 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6"] Jul 25 01:22:48.481075 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.481035 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3caf0a5e-8c56-468e-85bb-7c1581d43618-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-shjr6\" (UID: \"3caf0a5e-8c56-468e-85bb-7c1581d43618\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.481265 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.481101 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3caf0a5e-8c56-468e-85bb-7c1581d43618-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-shjr6\" (UID: \"3caf0a5e-8c56-468e-85bb-7c1581d43618\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.481265 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.481175 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d2gx\" (UniqueName: \"kubernetes.io/projected/3caf0a5e-8c56-468e-85bb-7c1581d43618-kube-api-access-8d2gx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-shjr6\" (UID: \"3caf0a5e-8c56-468e-85bb-7c1581d43618\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.481265 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.481245 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3caf0a5e-8c56-468e-85bb-7c1581d43618-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-shjr6\" (UID: \"3caf0a5e-8c56-468e-85bb-7c1581d43618\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.481432 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.481306 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3caf0a5e-8c56-468e-85bb-7c1581d43618-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-shjr6\" (UID: \"3caf0a5e-8c56-468e-85bb-7c1581d43618\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.481432 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.481342 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3caf0a5e-8c56-468e-85bb-7c1581d43618-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-shjr6\" (UID: \"3caf0a5e-8c56-468e-85bb-7c1581d43618\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.582510 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.582419 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3caf0a5e-8c56-468e-85bb-7c1581d43618-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-shjr6\" (UID: \"3caf0a5e-8c56-468e-85bb-7c1581d43618\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.582510 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.582484 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3caf0a5e-8c56-468e-85bb-7c1581d43618-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-shjr6\" (UID: \"3caf0a5e-8c56-468e-85bb-7c1581d43618\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.582718 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.582646 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d2gx\" (UniqueName: \"kubernetes.io/projected/3caf0a5e-8c56-468e-85bb-7c1581d43618-kube-api-access-8d2gx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-shjr6\" (UID: \"3caf0a5e-8c56-468e-85bb-7c1581d43618\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.582718 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.582680 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3caf0a5e-8c56-468e-85bb-7c1581d43618-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-shjr6\" (UID: \"3caf0a5e-8c56-468e-85bb-7c1581d43618\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.582849 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.582721 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3caf0a5e-8c56-468e-85bb-7c1581d43618-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-shjr6\" (UID: \"3caf0a5e-8c56-468e-85bb-7c1581d43618\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.582849 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.582752 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3caf0a5e-8c56-468e-85bb-7c1581d43618-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-shjr6\" (UID: \"3caf0a5e-8c56-468e-85bb-7c1581d43618\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.582966 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.582949 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3caf0a5e-8c56-468e-85bb-7c1581d43618-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-shjr6\" (UID: \"3caf0a5e-8c56-468e-85bb-7c1581d43618\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.583071 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.583048 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3caf0a5e-8c56-468e-85bb-7c1581d43618-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-shjr6\" (UID: \"3caf0a5e-8c56-468e-85bb-7c1581d43618\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.583163 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.583142 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3caf0a5e-8c56-468e-85bb-7c1581d43618-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-shjr6\" (UID: \"3caf0a5e-8c56-468e-85bb-7c1581d43618\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.584817 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.584777 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3caf0a5e-8c56-468e-85bb-7c1581d43618-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-shjr6\" (UID: \"3caf0a5e-8c56-468e-85bb-7c1581d43618\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.585422 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.585399 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3caf0a5e-8c56-468e-85bb-7c1581d43618-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-shjr6\" (UID: \"3caf0a5e-8c56-468e-85bb-7c1581d43618\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.591994 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.591973 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d2gx\" (UniqueName: \"kubernetes.io/projected/3caf0a5e-8c56-468e-85bb-7c1581d43618-kube-api-access-8d2gx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-shjr6\" (UID: \"3caf0a5e-8c56-468e-85bb-7c1581d43618\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.640985 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.640964 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:48.776948 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:48.776919 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6"] Jul 25 01:22:48.779029 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:22:48.779003 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3caf0a5e_8c56_468e_85bb_7c1581d43618.slice/crio-a958a5b6d0fa96f986df19f9b7f32321ec9a9fb2a18cb08045c3baee5d5197f6 WatchSource:0}: Error finding container a958a5b6d0fa96f986df19f9b7f32321ec9a9fb2a18cb08045c3baee5d5197f6: Status 404 returned error can't find the container with id a958a5b6d0fa96f986df19f9b7f32321ec9a9fb2a18cb08045c3baee5d5197f6 Jul 25 01:22:49.663401 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:49.663320 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" event={"ID":"3caf0a5e-8c56-468e-85bb-7c1581d43618","Type":"ContainerStarted","Data":"d1e43f75c0052a2e2e27af852c25c26534e332cea89434938e2deb8da66f0094"} Jul 25 01:22:49.663814 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:49.663401 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" event={"ID":"3caf0a5e-8c56-468e-85bb-7c1581d43618","Type":"ContainerStarted","Data":"a958a5b6d0fa96f986df19f9b7f32321ec9a9fb2a18cb08045c3baee5d5197f6"} Jul 25 01:22:49.680173 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:49.680131 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" podStartSLOduration=1.455447306 podStartE2EDuration="1.680116718s" totalImagesPullingTime="224.669412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:22:49.677710509 +0000 UTC m=+710.799196359" watchObservedRunningTime="2026-07-25 01:22:49.680116718 +0000 UTC m=+710.801602568" Jul 25 01:22:58.641233 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:58.641197 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:58.641628 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:58.641285 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:58.653830 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:58.653786 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:22:58.725586 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:22:58.725553 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-shjr6" Jul 25 01:30:47.672938 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:30:47.672889 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb"] Jul 25 01:30:47.673531 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:30:47.673122 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb" podUID="c0cfc251-80e6-43a5-9155-c0563bd1dd61" containerName="manager" containerID="cri-o://aeea9114e099cf7c2e01a1f231ae6f694c85b7c275839431e6f94e98a46f9028" gracePeriod=10 Jul 25 01:30:50.532577 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:30:50.532552 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb" Jul 25 01:30:50.681088 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:30:50.681061 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0cfc251-80e6-43a5-9155-c0563bd1dd61-extensions-socket-volume\") pod \"c0cfc251-80e6-43a5-9155-c0563bd1dd61\" (UID: \"c0cfc251-80e6-43a5-9155-c0563bd1dd61\") " Jul 25 01:30:50.681237 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:30:50.681100 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wvdh\" (UniqueName: \"kubernetes.io/projected/c0cfc251-80e6-43a5-9155-c0563bd1dd61-kube-api-access-7wvdh\") pod \"c0cfc251-80e6-43a5-9155-c0563bd1dd61\" (UID: \"c0cfc251-80e6-43a5-9155-c0563bd1dd61\") " Jul 25 01:30:50.681425 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:30:50.681397 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cfc251-80e6-43a5-9155-c0563bd1dd61-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c0cfc251-80e6-43a5-9155-c0563bd1dd61" (UID: "c0cfc251-80e6-43a5-9155-c0563bd1dd61"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 01:30:50.683305 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:30:50.683285 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0cfc251-80e6-43a5-9155-c0563bd1dd61-kube-api-access-7wvdh" (OuterVolumeSpecName: "kube-api-access-7wvdh") pod "c0cfc251-80e6-43a5-9155-c0563bd1dd61" (UID: "c0cfc251-80e6-43a5-9155-c0563bd1dd61"). InnerVolumeSpecName "kube-api-access-7wvdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:30:50.781872 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:30:50.781837 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0cfc251-80e6-43a5-9155-c0563bd1dd61-extensions-socket-volume\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:30:50.781872 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:30:50.781868 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wvdh\" (UniqueName: \"kubernetes.io/projected/c0cfc251-80e6-43a5-9155-c0563bd1dd61-kube-api-access-7wvdh\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:30:51.190748 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:30:51.190716 2554 generic.go:356] "Generic (PLEG): container finished" podID="c0cfc251-80e6-43a5-9155-c0563bd1dd61" containerID="aeea9114e099cf7c2e01a1f231ae6f694c85b7c275839431e6f94e98a46f9028" exitCode=0 Jul 25 01:30:51.190936 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:30:51.190765 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb" Jul 25 01:30:51.190936 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:30:51.190765 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb" event={"ID":"c0cfc251-80e6-43a5-9155-c0563bd1dd61","Type":"ContainerDied","Data":"aeea9114e099cf7c2e01a1f231ae6f694c85b7c275839431e6f94e98a46f9028"} Jul 25 01:30:51.190936 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:30:51.190839 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb" event={"ID":"c0cfc251-80e6-43a5-9155-c0563bd1dd61","Type":"ContainerDied","Data":"a1cacfc500ffd0ee6e4dbbd2b0a3d70a33ff48a3684b2a1ccc4f46920bc79e72"} Jul 25 01:30:51.190936 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:30:51.190865 2554 scope.go:117] "RemoveContainer" containerID="aeea9114e099cf7c2e01a1f231ae6f694c85b7c275839431e6f94e98a46f9028" Jul 25 01:30:51.201448 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:30:51.201431 2554 scope.go:117] "RemoveContainer" containerID="aeea9114e099cf7c2e01a1f231ae6f694c85b7c275839431e6f94e98a46f9028" Jul 25 01:30:51.201750 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:30:51.201730 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeea9114e099cf7c2e01a1f231ae6f694c85b7c275839431e6f94e98a46f9028\": container with ID starting with aeea9114e099cf7c2e01a1f231ae6f694c85b7c275839431e6f94e98a46f9028 not found: ID does not exist" containerID="aeea9114e099cf7c2e01a1f231ae6f694c85b7c275839431e6f94e98a46f9028" Jul 25 01:30:51.201831 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:30:51.201761 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeea9114e099cf7c2e01a1f231ae6f694c85b7c275839431e6f94e98a46f9028"} err="failed to get container status \"aeea9114e099cf7c2e01a1f231ae6f694c85b7c275839431e6f94e98a46f9028\": rpc error: code = NotFound desc = could not find container \"aeea9114e099cf7c2e01a1f231ae6f694c85b7c275839431e6f94e98a46f9028\": container with ID starting with aeea9114e099cf7c2e01a1f231ae6f694c85b7c275839431e6f94e98a46f9028 not found: ID does not exist" Jul 25 01:30:51.211160 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:30:51.211140 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb"] Jul 25 01:30:51.214784 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:30:51.214764 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2qfpb"] Jul 25 01:30:51.380783 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:30:51.380754 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0cfc251-80e6-43a5-9155-c0563bd1dd61" path="/var/lib/kubelet/pods/c0cfc251-80e6-43a5-9155-c0563bd1dd61/volumes" Jul 25 01:31:48.812180 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:31:48.812146 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d56jb"] Jul 25 01:31:48.814708 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:31:48.812661 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0cfc251-80e6-43a5-9155-c0563bd1dd61" containerName="manager" Jul 25 01:31:48.814708 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:31:48.812675 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0cfc251-80e6-43a5-9155-c0563bd1dd61" containerName="manager" Jul 25 01:31:48.814708 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:31:48.812736 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0cfc251-80e6-43a5-9155-c0563bd1dd61" containerName="manager" Jul 25 01:31:48.815602 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:31:48.815584 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d56jb" Jul 25 01:31:48.818232 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:31:48.818215 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lxtfb\"" Jul 25 01:31:48.827938 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:31:48.827892 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d56jb"] Jul 25 01:31:48.842179 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:31:48.842154 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd9sk\" (UniqueName: \"kubernetes.io/projected/8db6dc0d-58fa-4c88-a142-9408e7192967-kube-api-access-rd9sk\") pod \"kuadrant-operator-controller-manager-86878c769f-d56jb\" (UID: \"8db6dc0d-58fa-4c88-a142-9408e7192967\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d56jb" Jul 25 01:31:48.842286 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:31:48.842192 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8db6dc0d-58fa-4c88-a142-9408e7192967-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-d56jb\" (UID: \"8db6dc0d-58fa-4c88-a142-9408e7192967\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d56jb" Jul 25 01:31:48.943345 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:31:48.943316 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd9sk\" (UniqueName: \"kubernetes.io/projected/8db6dc0d-58fa-4c88-a142-9408e7192967-kube-api-access-rd9sk\") pod \"kuadrant-operator-controller-manager-86878c769f-d56jb\" (UID: \"8db6dc0d-58fa-4c88-a142-9408e7192967\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d56jb" Jul 25 01:31:48.943507 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:31:48.943354 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8db6dc0d-58fa-4c88-a142-9408e7192967-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-d56jb\" (UID: \"8db6dc0d-58fa-4c88-a142-9408e7192967\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d56jb" Jul 25 01:31:48.943687 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:31:48.943670 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8db6dc0d-58fa-4c88-a142-9408e7192967-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-d56jb\" (UID: \"8db6dc0d-58fa-4c88-a142-9408e7192967\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d56jb" Jul 25 01:31:48.951563 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:31:48.951542 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd9sk\" (UniqueName: \"kubernetes.io/projected/8db6dc0d-58fa-4c88-a142-9408e7192967-kube-api-access-rd9sk\") pod \"kuadrant-operator-controller-manager-86878c769f-d56jb\" (UID: \"8db6dc0d-58fa-4c88-a142-9408e7192967\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d56jb" Jul 25 01:31:49.126604 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:31:49.126521 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d56jb" Jul 25 01:31:49.266728 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:31:49.266706 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d56jb"] Jul 25 01:31:49.269207 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:31:49.269171 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db6dc0d_58fa_4c88_a142_9408e7192967.slice/crio-0460f4f52c2c8b4bcb94aef759efde83ff7659c71a922757744ba6bc5fa621c2 WatchSource:0}: Error finding container 0460f4f52c2c8b4bcb94aef759efde83ff7659c71a922757744ba6bc5fa621c2: Status 404 returned error can't find the container with id 0460f4f52c2c8b4bcb94aef759efde83ff7659c71a922757744ba6bc5fa621c2 Jul 25 01:31:49.490716 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:31:49.490687 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d56jb" event={"ID":"8db6dc0d-58fa-4c88-a142-9408e7192967","Type":"ContainerStarted","Data":"3f319bf9661cf5507562e44af85cb302a7bb5996562d842b9c10b2cddadf0c23"} Jul 25 01:31:49.490898 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:31:49.490733 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d56jb" event={"ID":"8db6dc0d-58fa-4c88-a142-9408e7192967","Type":"ContainerStarted","Data":"0460f4f52c2c8b4bcb94aef759efde83ff7659c71a922757744ba6bc5fa621c2"} Jul 25 01:31:49.490898 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:31:49.490773 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d56jb" Jul 25 01:31:49.509041 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:31:49.508989 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d56jb" podStartSLOduration=1.508972507 podStartE2EDuration="1.508972507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:31:49.506602623 +0000 UTC m=+1250.628088474" watchObservedRunningTime="2026-07-25 01:31:49.508972507 +0000 UTC m=+1250.630458356" Jul 25 01:32:00.497872 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:32:00.497844 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d56jb" Jul 25 01:38:57.634186 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:57.634156 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx"] Jul 25 01:38:57.637875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:57.637859 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" Jul 25 01:38:57.640062 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:57.640031 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-04dad736-serving-cert\"" Jul 25 01:38:57.649600 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:57.649575 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx"] Jul 25 01:38:57.757248 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:57.757219 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a23018-3a1f-4b73-b3b6-712086cacf5d-openshift-service-ca\") pod \"maas-api-e2e-ait-04dad736-67bc89868c-94pmx\" (UID: \"33a23018-3a1f-4b73-b3b6-712086cacf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" Jul 25 01:38:57.757444 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:57.757255 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxpwn\" (UniqueName: \"kubernetes.io/projected/33a23018-3a1f-4b73-b3b6-712086cacf5d-kube-api-access-dxpwn\") pod \"maas-api-e2e-ait-04dad736-67bc89868c-94pmx\" (UID: \"33a23018-3a1f-4b73-b3b6-712086cacf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" Jul 25 01:38:57.757444 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:57.757274 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33a23018-3a1f-4b73-b3b6-712086cacf5d-maas-api-tls\") pod \"maas-api-e2e-ait-04dad736-67bc89868c-94pmx\" (UID: \"33a23018-3a1f-4b73-b3b6-712086cacf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" Jul 25 01:38:57.858397 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:57.858362 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a23018-3a1f-4b73-b3b6-712086cacf5d-openshift-service-ca\") pod \"maas-api-e2e-ait-04dad736-67bc89868c-94pmx\" (UID: \"33a23018-3a1f-4b73-b3b6-712086cacf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" Jul 25 01:38:57.858397 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:57.858399 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxpwn\" (UniqueName: \"kubernetes.io/projected/33a23018-3a1f-4b73-b3b6-712086cacf5d-kube-api-access-dxpwn\") pod \"maas-api-e2e-ait-04dad736-67bc89868c-94pmx\" (UID: \"33a23018-3a1f-4b73-b3b6-712086cacf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" Jul 25 01:38:57.858641 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:57.858422 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33a23018-3a1f-4b73-b3b6-712086cacf5d-maas-api-tls\") pod \"maas-api-e2e-ait-04dad736-67bc89868c-94pmx\" (UID: \"33a23018-3a1f-4b73-b3b6-712086cacf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" Jul 25 01:38:57.858641 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:38:57.858519 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-serving-cert: secret "maas-api-e2e-ait-04dad736-serving-cert" not found Jul 25 01:38:57.858641 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:38:57.858593 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33a23018-3a1f-4b73-b3b6-712086cacf5d-maas-api-tls podName:33a23018-3a1f-4b73-b3b6-712086cacf5d nodeName:}" failed. No retries permitted until 2026-07-25 01:38:58.358570178 +0000 UTC m=+1679.480056009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33a23018-3a1f-4b73-b3b6-712086cacf5d-maas-api-tls") pod "maas-api-e2e-ait-04dad736-67bc89868c-94pmx" (UID: "33a23018-3a1f-4b73-b3b6-712086cacf5d") : secret "maas-api-e2e-ait-04dad736-serving-cert" not found Jul 25 01:38:57.859114 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:57.859093 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a23018-3a1f-4b73-b3b6-712086cacf5d-openshift-service-ca\") pod \"maas-api-e2e-ait-04dad736-67bc89868c-94pmx\" (UID: \"33a23018-3a1f-4b73-b3b6-712086cacf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" Jul 25 01:38:57.873312 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:57.873290 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxpwn\" (UniqueName: \"kubernetes.io/projected/33a23018-3a1f-4b73-b3b6-712086cacf5d-kube-api-access-dxpwn\") pod \"maas-api-e2e-ait-04dad736-67bc89868c-94pmx\" (UID: \"33a23018-3a1f-4b73-b3b6-712086cacf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" Jul 25 01:38:58.363971 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:58.363940 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33a23018-3a1f-4b73-b3b6-712086cacf5d-maas-api-tls\") pod \"maas-api-e2e-ait-04dad736-67bc89868c-94pmx\" (UID: \"33a23018-3a1f-4b73-b3b6-712086cacf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" Jul 25 01:38:58.366440 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:58.366420 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33a23018-3a1f-4b73-b3b6-712086cacf5d-maas-api-tls\") pod \"maas-api-e2e-ait-04dad736-67bc89868c-94pmx\" (UID: \"33a23018-3a1f-4b73-b3b6-712086cacf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" Jul 25 01:38:58.551600 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:58.551555 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" Jul 25 01:38:58.690671 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:58.690640 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx"] Jul 25 01:38:58.692178 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:38:58.692153 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a23018_3a1f_4b73_b3b6_712086cacf5d.slice/crio-4abc7437a2443c82665de8e48c7a0dab9d53e19a6f5a4e9b9a266e66dc8d76f1 WatchSource:0}: Error finding container 4abc7437a2443c82665de8e48c7a0dab9d53e19a6f5a4e9b9a266e66dc8d76f1: Status 404 returned error can't find the container with id 4abc7437a2443c82665de8e48c7a0dab9d53e19a6f5a4e9b9a266e66dc8d76f1 Jul 25 01:38:58.693906 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:58.693889 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 25 01:38:58.722570 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:58.722536 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" event={"ID":"33a23018-3a1f-4b73-b3b6-712086cacf5d","Type":"ContainerStarted","Data":"4abc7437a2443c82665de8e48c7a0dab9d53e19a6f5a4e9b9a266e66dc8d76f1"} Jul 25 01:38:59.729591 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:59.729548 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" event={"ID":"33a23018-3a1f-4b73-b3b6-712086cacf5d","Type":"ContainerStarted","Data":"cf44d3c126fe99dc2725c09be48df2637ef18069cc697e9c99f4bed495ad84f3"} Jul 25 01:38:59.730111 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:59.729689 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" Jul 25 01:38:59.744971 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:38:59.744931 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" podStartSLOduration=2.269605207 podStartE2EDuration="2.744917647s" totalImagesPullingTime="475.31244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:38:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:38:59.743980711 +0000 UTC m=+1680.865466561" watchObservedRunningTime="2026-07-25 01:38:59.744917647 +0000 UTC m=+1680.866403496" Jul 25 01:39:05.741266 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:39:05.741236 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" Jul 25 01:44:14.520120 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:14.520082 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx"] Jul 25 01:44:14.520668 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:14.520352 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" podUID="33a23018-3a1f-4b73-b3b6-712086cacf5d" containerName="maas-api" containerID="cri-o://cf44d3c126fe99dc2725c09be48df2637ef18069cc697e9c99f4bed495ad84f3" gracePeriod=30 Jul 25 01:44:14.780121 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:14.780055 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" Jul 25 01:44:14.846704 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:14.846674 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a23018-3a1f-4b73-b3b6-712086cacf5d-openshift-service-ca\") pod \"33a23018-3a1f-4b73-b3b6-712086cacf5d\" (UID: \"33a23018-3a1f-4b73-b3b6-712086cacf5d\") " Jul 25 01:44:14.846875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:14.846712 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33a23018-3a1f-4b73-b3b6-712086cacf5d-maas-api-tls\") pod \"33a23018-3a1f-4b73-b3b6-712086cacf5d\" (UID: \"33a23018-3a1f-4b73-b3b6-712086cacf5d\") " Jul 25 01:44:14.846875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:14.846735 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxpwn\" (UniqueName: \"kubernetes.io/projected/33a23018-3a1f-4b73-b3b6-712086cacf5d-kube-api-access-dxpwn\") pod \"33a23018-3a1f-4b73-b3b6-712086cacf5d\" (UID: \"33a23018-3a1f-4b73-b3b6-712086cacf5d\") " Jul 25 01:44:14.847064 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:14.847043 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a23018-3a1f-4b73-b3b6-712086cacf5d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33a23018-3a1f-4b73-b3b6-712086cacf5d" (UID: "33a23018-3a1f-4b73-b3b6-712086cacf5d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 01:44:14.849128 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:14.849097 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a23018-3a1f-4b73-b3b6-712086cacf5d-kube-api-access-dxpwn" (OuterVolumeSpecName: "kube-api-access-dxpwn") pod "33a23018-3a1f-4b73-b3b6-712086cacf5d" (UID: "33a23018-3a1f-4b73-b3b6-712086cacf5d"). InnerVolumeSpecName "kube-api-access-dxpwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:44:14.849247 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:14.849136 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a23018-3a1f-4b73-b3b6-712086cacf5d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33a23018-3a1f-4b73-b3b6-712086cacf5d" (UID: "33a23018-3a1f-4b73-b3b6-712086cacf5d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 01:44:14.947331 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:14.947303 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a23018-3a1f-4b73-b3b6-712086cacf5d-openshift-service-ca\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:44:14.947331 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:14.947328 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33a23018-3a1f-4b73-b3b6-712086cacf5d-maas-api-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:44:14.947331 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:14.947337 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxpwn\" (UniqueName: \"kubernetes.io/projected/33a23018-3a1f-4b73-b3b6-712086cacf5d-kube-api-access-dxpwn\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:44:15.390112 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:15.390083 2554 generic.go:356] "Generic (PLEG): container finished" podID="33a23018-3a1f-4b73-b3b6-712086cacf5d" containerID="cf44d3c126fe99dc2725c09be48df2637ef18069cc697e9c99f4bed495ad84f3" exitCode=0 Jul 25 01:44:15.390262 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:15.390119 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" Jul 25 01:44:15.390262 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:15.390173 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" event={"ID":"33a23018-3a1f-4b73-b3b6-712086cacf5d","Type":"ContainerDied","Data":"cf44d3c126fe99dc2725c09be48df2637ef18069cc697e9c99f4bed495ad84f3"} Jul 25 01:44:15.390262 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:15.390230 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx" event={"ID":"33a23018-3a1f-4b73-b3b6-712086cacf5d","Type":"ContainerDied","Data":"4abc7437a2443c82665de8e48c7a0dab9d53e19a6f5a4e9b9a266e66dc8d76f1"} Jul 25 01:44:15.390262 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:15.390258 2554 scope.go:117] "RemoveContainer" containerID="cf44d3c126fe99dc2725c09be48df2637ef18069cc697e9c99f4bed495ad84f3" Jul 25 01:44:15.399516 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:15.399498 2554 scope.go:117] "RemoveContainer" containerID="cf44d3c126fe99dc2725c09be48df2637ef18069cc697e9c99f4bed495ad84f3" Jul 25 01:44:15.399870 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:44:15.399827 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf44d3c126fe99dc2725c09be48df2637ef18069cc697e9c99f4bed495ad84f3\": container with ID starting with cf44d3c126fe99dc2725c09be48df2637ef18069cc697e9c99f4bed495ad84f3 not found: ID does not exist" containerID="cf44d3c126fe99dc2725c09be48df2637ef18069cc697e9c99f4bed495ad84f3" Jul 25 01:44:15.399955 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:15.399876 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf44d3c126fe99dc2725c09be48df2637ef18069cc697e9c99f4bed495ad84f3"} err="failed to get container status \"cf44d3c126fe99dc2725c09be48df2637ef18069cc697e9c99f4bed495ad84f3\": rpc error: code = NotFound desc = could not find container \"cf44d3c126fe99dc2725c09be48df2637ef18069cc697e9c99f4bed495ad84f3\": container with ID starting with cf44d3c126fe99dc2725c09be48df2637ef18069cc697e9c99f4bed495ad84f3 not found: ID does not exist" Jul 25 01:44:15.404409 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:15.404389 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx"] Jul 25 01:44:15.410087 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:15.410068 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-04dad736-67bc89868c-94pmx"] Jul 25 01:44:17.381579 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:17.381549 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33a23018-3a1f-4b73-b3b6-712086cacf5d" path="/var/lib/kubelet/pods/33a23018-3a1f-4b73-b3b6-712086cacf5d/volumes" Jul 25 01:44:22.535302 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:22.535268 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n"] Jul 25 01:44:22.536297 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:22.536271 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33a23018-3a1f-4b73-b3b6-712086cacf5d" containerName="maas-api" Jul 25 01:44:22.536297 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:22.536296 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="33a23018-3a1f-4b73-b3b6-712086cacf5d" containerName="maas-api" Jul 25 01:44:22.536505 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:22.536355 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="33a23018-3a1f-4b73-b3b6-712086cacf5d" containerName="maas-api" Jul 25 01:44:22.540834 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:22.540813 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" Jul 25 01:44:22.542740 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:22.542719 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-db7d7068-serving-cert\"" Jul 25 01:44:22.551378 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:22.551356 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n"] Jul 25 01:44:22.620651 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:22.620615 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23280ea6-f43e-4736-a355-c15d0f5a0327-maas-api-tls\") pod \"maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n\" (UID: \"23280ea6-f43e-4736-a355-c15d0f5a0327\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" Jul 25 01:44:22.620861 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:22.620660 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrg65\" (UniqueName: \"kubernetes.io/projected/23280ea6-f43e-4736-a355-c15d0f5a0327-kube-api-access-qrg65\") pod \"maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n\" (UID: \"23280ea6-f43e-4736-a355-c15d0f5a0327\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" Jul 25 01:44:22.620861 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:22.620781 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23280ea6-f43e-4736-a355-c15d0f5a0327-openshift-service-ca\") pod \"maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n\" (UID: \"23280ea6-f43e-4736-a355-c15d0f5a0327\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" Jul 25 01:44:22.721972 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:22.721935 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23280ea6-f43e-4736-a355-c15d0f5a0327-maas-api-tls\") pod \"maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n\" (UID: \"23280ea6-f43e-4736-a355-c15d0f5a0327\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" Jul 25 01:44:22.722140 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:22.722007 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrg65\" (UniqueName: \"kubernetes.io/projected/23280ea6-f43e-4736-a355-c15d0f5a0327-kube-api-access-qrg65\") pod \"maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n\" (UID: \"23280ea6-f43e-4736-a355-c15d0f5a0327\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" Jul 25 01:44:22.722140 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:22.722110 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23280ea6-f43e-4736-a355-c15d0f5a0327-openshift-service-ca\") pod \"maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n\" (UID: \"23280ea6-f43e-4736-a355-c15d0f5a0327\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" Jul 25 01:44:22.722845 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:22.722818 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23280ea6-f43e-4736-a355-c15d0f5a0327-openshift-service-ca\") pod \"maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n\" (UID: \"23280ea6-f43e-4736-a355-c15d0f5a0327\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" Jul 25 01:44:22.724747 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:22.724730 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23280ea6-f43e-4736-a355-c15d0f5a0327-maas-api-tls\") pod \"maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n\" (UID: \"23280ea6-f43e-4736-a355-c15d0f5a0327\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" Jul 25 01:44:22.729452 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:22.729429 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrg65\" (UniqueName: \"kubernetes.io/projected/23280ea6-f43e-4736-a355-c15d0f5a0327-kube-api-access-qrg65\") pod \"maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n\" (UID: \"23280ea6-f43e-4736-a355-c15d0f5a0327\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" Jul 25 01:44:22.854586 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:22.854497 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" Jul 25 01:44:22.995020 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:22.994959 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n"] Jul 25 01:44:22.996217 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:44:22.996180 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23280ea6_f43e_4736_a355_c15d0f5a0327.slice/crio-b0ae06eeae97fc6aaac536aba08179ed3bba0c48f567b09d68bf4910972c5ff7 WatchSource:0}: Error finding container b0ae06eeae97fc6aaac536aba08179ed3bba0c48f567b09d68bf4910972c5ff7: Status 404 returned error can't find the container with id b0ae06eeae97fc6aaac536aba08179ed3bba0c48f567b09d68bf4910972c5ff7 Jul 25 01:44:22.997670 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:22.997648 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 25 01:44:23.324055 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.324020 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4"] Jul 25 01:44:23.328524 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.328498 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4" Jul 25 01:44:23.330781 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.330759 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-db7d7068\"" Jul 25 01:44:23.330911 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.330867 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-db7d7068-dockercfg-x8hkb\"" Jul 25 01:44:23.336223 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.336158 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4"] Jul 25 01:44:23.427991 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.427959 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/776baa6a-afd7-4f8b-9757-9bd7a6fee63b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4\" (UID: \"776baa6a-afd7-4f8b-9757-9bd7a6fee63b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4" Jul 25 01:44:23.428157 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.428062 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc9rv\" (UniqueName: \"kubernetes.io/projected/776baa6a-afd7-4f8b-9757-9bd7a6fee63b-kube-api-access-tc9rv\") pod \"payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4\" (UID: \"776baa6a-afd7-4f8b-9757-9bd7a6fee63b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4" Jul 25 01:44:23.434240 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.434213 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" event={"ID":"23280ea6-f43e-4736-a355-c15d0f5a0327","Type":"ContainerStarted","Data":"b0ae06eeae97fc6aaac536aba08179ed3bba0c48f567b09d68bf4910972c5ff7"} Jul 25 01:44:23.469214 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.469182 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r"] Jul 25 01:44:23.473029 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.473014 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r" Jul 25 01:44:23.483291 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.483266 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r"] Jul 25 01:44:23.528639 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.528611 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/776baa6a-afd7-4f8b-9757-9bd7a6fee63b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4\" (UID: \"776baa6a-afd7-4f8b-9757-9bd7a6fee63b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4" Jul 25 01:44:23.528768 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.528662 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc9rv\" (UniqueName: \"kubernetes.io/projected/776baa6a-afd7-4f8b-9757-9bd7a6fee63b-kube-api-access-tc9rv\") pod \"payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4\" (UID: \"776baa6a-afd7-4f8b-9757-9bd7a6fee63b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4" Jul 25 01:44:23.529307 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.529287 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/776baa6a-afd7-4f8b-9757-9bd7a6fee63b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4\" (UID: \"776baa6a-afd7-4f8b-9757-9bd7a6fee63b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4" Jul 25 01:44:23.536381 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.536354 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc9rv\" (UniqueName: \"kubernetes.io/projected/776baa6a-afd7-4f8b-9757-9bd7a6fee63b-kube-api-access-tc9rv\") pod \"payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4\" (UID: \"776baa6a-afd7-4f8b-9757-9bd7a6fee63b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4" Jul 25 01:44:23.629546 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.629464 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrw4c\" (UniqueName: \"kubernetes.io/projected/331f599b-57be-4365-af8b-51fa1653e35a-kube-api-access-zrw4c\") pod \"payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r\" (UID: \"331f599b-57be-4365-af8b-51fa1653e35a\") " pod="openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r" Jul 25 01:44:23.629546 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.629505 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/331f599b-57be-4365-af8b-51fa1653e35a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r\" (UID: \"331f599b-57be-4365-af8b-51fa1653e35a\") " pod="openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r" Jul 25 01:44:23.640319 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.640290 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4" Jul 25 01:44:23.730419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.730384 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrw4c\" (UniqueName: \"kubernetes.io/projected/331f599b-57be-4365-af8b-51fa1653e35a-kube-api-access-zrw4c\") pod \"payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r\" (UID: \"331f599b-57be-4365-af8b-51fa1653e35a\") " pod="openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r" Jul 25 01:44:23.730565 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.730459 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/331f599b-57be-4365-af8b-51fa1653e35a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r\" (UID: \"331f599b-57be-4365-af8b-51fa1653e35a\") " pod="openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r" Jul 25 01:44:23.731490 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.731466 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/331f599b-57be-4365-af8b-51fa1653e35a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r\" (UID: \"331f599b-57be-4365-af8b-51fa1653e35a\") " pod="openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r" Jul 25 01:44:23.743637 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.743612 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrw4c\" (UniqueName: \"kubernetes.io/projected/331f599b-57be-4365-af8b-51fa1653e35a-kube-api-access-zrw4c\") pod \"payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r\" (UID: \"331f599b-57be-4365-af8b-51fa1653e35a\") " pod="openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r" Jul 25 01:44:23.784204 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.784179 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r" Jul 25 01:44:23.790575 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:23.790555 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4"] Jul 25 01:44:23.795229 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:44:23.795193 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776baa6a_afd7_4f8b_9757_9bd7a6fee63b.slice/crio-a3f87eee8d0624e31aead95f3fd5af8321a9f3d8c38243c44814558a4be49936 WatchSource:0}: Error finding container a3f87eee8d0624e31aead95f3fd5af8321a9f3d8c38243c44814558a4be49936: Status 404 returned error can't find the container with id a3f87eee8d0624e31aead95f3fd5af8321a9f3d8c38243c44814558a4be49936 Jul 25 01:44:24.140733 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:24.140693 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r"] Jul 25 01:44:24.142655 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:44:24.142631 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod331f599b_57be_4365_af8b_51fa1653e35a.slice/crio-1a6f45e05232aed46e4217ebfc6952376794801e9ec65e31888a423e1536d301 WatchSource:0}: Error finding container 1a6f45e05232aed46e4217ebfc6952376794801e9ec65e31888a423e1536d301: Status 404 returned error can't find the container with id 1a6f45e05232aed46e4217ebfc6952376794801e9ec65e31888a423e1536d301 Jul 25 01:44:24.441789 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:24.441751 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r" event={"ID":"331f599b-57be-4365-af8b-51fa1653e35a","Type":"ContainerStarted","Data":"9bcbfec69cc287ef899a6831b794d5030c0e36e6e4e51d70fb326de8064b9bd5"} Jul 25 01:44:24.442000 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:24.441978 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r" Jul 25 01:44:24.442100 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:24.442027 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r" event={"ID":"331f599b-57be-4365-af8b-51fa1653e35a","Type":"ContainerStarted","Data":"1a6f45e05232aed46e4217ebfc6952376794801e9ec65e31888a423e1536d301"} Jul 25 01:44:24.443356 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:24.443327 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4" event={"ID":"776baa6a-afd7-4f8b-9757-9bd7a6fee63b","Type":"ContainerStarted","Data":"79dc457f2d05955902f93027fcbe106cd437c2fcab34483512dd1cb45ada8192"} Jul 25 01:44:24.443507 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:24.443370 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4" event={"ID":"776baa6a-afd7-4f8b-9757-9bd7a6fee63b","Type":"ContainerStarted","Data":"a3f87eee8d0624e31aead95f3fd5af8321a9f3d8c38243c44814558a4be49936"} Jul 25 01:44:24.443507 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:24.443412 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4" Jul 25 01:44:24.444875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:24.444854 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" event={"ID":"23280ea6-f43e-4736-a355-c15d0f5a0327","Type":"ContainerStarted","Data":"cd42ba7cbf1f71b9c40c97bf1d55c986f399918237f30f6e9db917a1c11f5de5"} Jul 25 01:44:24.445004 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:24.444913 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" Jul 25 01:44:24.456410 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:24.456369 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r" podStartSLOduration=1.456353324 podStartE2EDuration="1.456353324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:44:24.45586248 +0000 UTC m=+2005.577348327" watchObservedRunningTime="2026-07-25 01:44:24.456353324 +0000 UTC m=+2005.577839174" Jul 25 01:44:24.471431 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:24.471148 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4" podStartSLOduration=1.471131421 podStartE2EDuration="1.471131421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:44:24.468910655 +0000 UTC m=+2005.590396505" watchObservedRunningTime="2026-07-25 01:44:24.471131421 +0000 UTC m=+2005.592617272" Jul 25 01:44:24.485830 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:24.485770 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" podStartSLOduration=1.766435828 podStartE2EDuration="2.485760215s" totalImagesPullingTime="719.324387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:44:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:44:24.482901001 +0000 UTC m=+2005.604386876" watchObservedRunningTime="2026-07-25 01:44:24.485760215 +0000 UTC m=+2005.607246065" Jul 25 01:44:30.455483 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:30.455454 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" Jul 25 01:44:35.452197 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:35.452167 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r" Jul 25 01:44:35.452627 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:44:35.452331 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4" Jul 25 01:47:41.627221 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:41.627186 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n"] Jul 25 01:47:41.627873 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:41.627795 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" podUID="23280ea6-f43e-4736-a355-c15d0f5a0327" containerName="maas-api" containerID="cri-o://cd42ba7cbf1f71b9c40c97bf1d55c986f399918237f30f6e9db917a1c11f5de5" gracePeriod=30 Jul 25 01:47:41.683077 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:41.683034 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r"] Jul 25 01:47:41.683327 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:41.683304 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r" podUID="331f599b-57be-4365-af8b-51fa1653e35a" containerName="payload-processing" containerID="cri-o://9bcbfec69cc287ef899a6831b794d5030c0e36e6e4e51d70fb326de8064b9bd5" gracePeriod=30 Jul 25 01:47:41.691647 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:41.691416 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4"] Jul 25 01:47:41.691784 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:41.691683 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4" podUID="776baa6a-afd7-4f8b-9757-9bd7a6fee63b" containerName="payload-pre-processing" containerID="cri-o://79dc457f2d05955902f93027fcbe106cd437c2fcab34483512dd1cb45ada8192" gracePeriod=30 Jul 25 01:47:41.955535 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:41.955505 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4" Jul 25 01:47:41.963554 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:41.963531 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" Jul 25 01:47:42.004685 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.004661 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r" Jul 25 01:47:42.079568 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.079535 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23280ea6-f43e-4736-a355-c15d0f5a0327-maas-api-tls\") pod \"23280ea6-f43e-4736-a355-c15d0f5a0327\" (UID: \"23280ea6-f43e-4736-a355-c15d0f5a0327\") " Jul 25 01:47:42.079761 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.079577 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrg65\" (UniqueName: \"kubernetes.io/projected/23280ea6-f43e-4736-a355-c15d0f5a0327-kube-api-access-qrg65\") pod \"23280ea6-f43e-4736-a355-c15d0f5a0327\" (UID: \"23280ea6-f43e-4736-a355-c15d0f5a0327\") " Jul 25 01:47:42.079761 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.079613 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tc9rv\" (UniqueName: \"kubernetes.io/projected/776baa6a-afd7-4f8b-9757-9bd7a6fee63b-kube-api-access-tc9rv\") pod \"776baa6a-afd7-4f8b-9757-9bd7a6fee63b\" (UID: \"776baa6a-afd7-4f8b-9757-9bd7a6fee63b\") " Jul 25 01:47:42.079761 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.079634 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrw4c\" (UniqueName: \"kubernetes.io/projected/331f599b-57be-4365-af8b-51fa1653e35a-kube-api-access-zrw4c\") pod \"331f599b-57be-4365-af8b-51fa1653e35a\" (UID: \"331f599b-57be-4365-af8b-51fa1653e35a\") " Jul 25 01:47:42.079761 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.079656 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/331f599b-57be-4365-af8b-51fa1653e35a-plugins-config-volume\") pod \"331f599b-57be-4365-af8b-51fa1653e35a\" (UID: \"331f599b-57be-4365-af8b-51fa1653e35a\") " Jul 25 01:47:42.079761 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.079690 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/776baa6a-afd7-4f8b-9757-9bd7a6fee63b-plugins-config-volume\") pod \"776baa6a-afd7-4f8b-9757-9bd7a6fee63b\" (UID: \"776baa6a-afd7-4f8b-9757-9bd7a6fee63b\") " Jul 25 01:47:42.080041 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.079841 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23280ea6-f43e-4736-a355-c15d0f5a0327-openshift-service-ca\") pod \"23280ea6-f43e-4736-a355-c15d0f5a0327\" (UID: \"23280ea6-f43e-4736-a355-c15d0f5a0327\") " Jul 25 01:47:42.080155 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.080122 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/331f599b-57be-4365-af8b-51fa1653e35a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "331f599b-57be-4365-af8b-51fa1653e35a" (UID: "331f599b-57be-4365-af8b-51fa1653e35a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 01:47:42.080226 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.080181 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776baa6a-afd7-4f8b-9757-9bd7a6fee63b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "776baa6a-afd7-4f8b-9757-9bd7a6fee63b" (UID: "776baa6a-afd7-4f8b-9757-9bd7a6fee63b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 01:47:42.080453 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.080425 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23280ea6-f43e-4736-a355-c15d0f5a0327-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "23280ea6-f43e-4736-a355-c15d0f5a0327" (UID: "23280ea6-f43e-4736-a355-c15d0f5a0327"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 01:47:42.082228 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.082199 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23280ea6-f43e-4736-a355-c15d0f5a0327-kube-api-access-qrg65" (OuterVolumeSpecName: "kube-api-access-qrg65") pod "23280ea6-f43e-4736-a355-c15d0f5a0327" (UID: "23280ea6-f43e-4736-a355-c15d0f5a0327"). InnerVolumeSpecName "kube-api-access-qrg65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:47:42.082430 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.082405 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23280ea6-f43e-4736-a355-c15d0f5a0327-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "23280ea6-f43e-4736-a355-c15d0f5a0327" (UID: "23280ea6-f43e-4736-a355-c15d0f5a0327"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 01:47:42.082430 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.082405 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331f599b-57be-4365-af8b-51fa1653e35a-kube-api-access-zrw4c" (OuterVolumeSpecName: "kube-api-access-zrw4c") pod "331f599b-57be-4365-af8b-51fa1653e35a" (UID: "331f599b-57be-4365-af8b-51fa1653e35a"). InnerVolumeSpecName "kube-api-access-zrw4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:47:42.082604 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.082502 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776baa6a-afd7-4f8b-9757-9bd7a6fee63b-kube-api-access-tc9rv" (OuterVolumeSpecName: "kube-api-access-tc9rv") pod "776baa6a-afd7-4f8b-9757-9bd7a6fee63b" (UID: "776baa6a-afd7-4f8b-9757-9bd7a6fee63b"). InnerVolumeSpecName "kube-api-access-tc9rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:47:42.181161 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.181122 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23280ea6-f43e-4736-a355-c15d0f5a0327-openshift-service-ca\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:47:42.181161 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.181154 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23280ea6-f43e-4736-a355-c15d0f5a0327-maas-api-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:47:42.181161 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.181165 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrg65\" (UniqueName: \"kubernetes.io/projected/23280ea6-f43e-4736-a355-c15d0f5a0327-kube-api-access-qrg65\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:47:42.181466 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.181174 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc9rv\" (UniqueName: \"kubernetes.io/projected/776baa6a-afd7-4f8b-9757-9bd7a6fee63b-kube-api-access-tc9rv\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:47:42.181466 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.181183 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrw4c\" (UniqueName: \"kubernetes.io/projected/331f599b-57be-4365-af8b-51fa1653e35a-kube-api-access-zrw4c\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:47:42.181466 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.181191 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/331f599b-57be-4365-af8b-51fa1653e35a-plugins-config-volume\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:47:42.181466 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.181199 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/776baa6a-afd7-4f8b-9757-9bd7a6fee63b-plugins-config-volume\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:47:42.495199 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.495110 2554 generic.go:356] "Generic (PLEG): container finished" podID="331f599b-57be-4365-af8b-51fa1653e35a" containerID="9bcbfec69cc287ef899a6831b794d5030c0e36e6e4e51d70fb326de8064b9bd5" exitCode=0 Jul 25 01:47:42.495199 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.495153 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r" Jul 25 01:47:42.495434 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.495196 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r" event={"ID":"331f599b-57be-4365-af8b-51fa1653e35a","Type":"ContainerDied","Data":"9bcbfec69cc287ef899a6831b794d5030c0e36e6e4e51d70fb326de8064b9bd5"} Jul 25 01:47:42.495434 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.495245 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r" event={"ID":"331f599b-57be-4365-af8b-51fa1653e35a","Type":"ContainerDied","Data":"1a6f45e05232aed46e4217ebfc6952376794801e9ec65e31888a423e1536d301"} Jul 25 01:47:42.495434 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.495268 2554 scope.go:117] "RemoveContainer" containerID="9bcbfec69cc287ef899a6831b794d5030c0e36e6e4e51d70fb326de8064b9bd5" Jul 25 01:47:42.496704 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.496605 2554 generic.go:356] "Generic (PLEG): container finished" podID="776baa6a-afd7-4f8b-9757-9bd7a6fee63b" containerID="79dc457f2d05955902f93027fcbe106cd437c2fcab34483512dd1cb45ada8192" exitCode=0 Jul 25 01:47:42.496704 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.496645 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4" Jul 25 01:47:42.496704 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.496683 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4" event={"ID":"776baa6a-afd7-4f8b-9757-9bd7a6fee63b","Type":"ContainerDied","Data":"79dc457f2d05955902f93027fcbe106cd437c2fcab34483512dd1cb45ada8192"} Jul 25 01:47:42.496976 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.496739 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4" event={"ID":"776baa6a-afd7-4f8b-9757-9bd7a6fee63b","Type":"ContainerDied","Data":"a3f87eee8d0624e31aead95f3fd5af8321a9f3d8c38243c44814558a4be49936"} Jul 25 01:47:42.498426 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.498400 2554 generic.go:356] "Generic (PLEG): container finished" podID="23280ea6-f43e-4736-a355-c15d0f5a0327" containerID="cd42ba7cbf1f71b9c40c97bf1d55c986f399918237f30f6e9db917a1c11f5de5" exitCode=0 Jul 25 01:47:42.498545 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.498444 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" Jul 25 01:47:42.498545 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.498464 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" event={"ID":"23280ea6-f43e-4736-a355-c15d0f5a0327","Type":"ContainerDied","Data":"cd42ba7cbf1f71b9c40c97bf1d55c986f399918237f30f6e9db917a1c11f5de5"} Jul 25 01:47:42.498545 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.498525 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n" event={"ID":"23280ea6-f43e-4736-a355-c15d0f5a0327","Type":"ContainerDied","Data":"b0ae06eeae97fc6aaac536aba08179ed3bba0c48f567b09d68bf4910972c5ff7"} Jul 25 01:47:42.507285 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.507265 2554 scope.go:117] "RemoveContainer" containerID="9bcbfec69cc287ef899a6831b794d5030c0e36e6e4e51d70fb326de8064b9bd5" Jul 25 01:47:42.507624 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:47:42.507577 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bcbfec69cc287ef899a6831b794d5030c0e36e6e4e51d70fb326de8064b9bd5\": container with ID starting with 9bcbfec69cc287ef899a6831b794d5030c0e36e6e4e51d70fb326de8064b9bd5 not found: ID does not exist" containerID="9bcbfec69cc287ef899a6831b794d5030c0e36e6e4e51d70fb326de8064b9bd5" Jul 25 01:47:42.507738 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.507620 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bcbfec69cc287ef899a6831b794d5030c0e36e6e4e51d70fb326de8064b9bd5"} err="failed to get container status \"9bcbfec69cc287ef899a6831b794d5030c0e36e6e4e51d70fb326de8064b9bd5\": rpc error: code = NotFound desc = could not find container \"9bcbfec69cc287ef899a6831b794d5030c0e36e6e4e51d70fb326de8064b9bd5\": container with ID starting with 9bcbfec69cc287ef899a6831b794d5030c0e36e6e4e51d70fb326de8064b9bd5 not found: ID does not exist" Jul 25 01:47:42.507738 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.507642 2554 scope.go:117] "RemoveContainer" containerID="79dc457f2d05955902f93027fcbe106cd437c2fcab34483512dd1cb45ada8192" Jul 25 01:47:42.516420 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.516406 2554 scope.go:117] "RemoveContainer" containerID="79dc457f2d05955902f93027fcbe106cd437c2fcab34483512dd1cb45ada8192" Jul 25 01:47:42.516666 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:47:42.516650 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79dc457f2d05955902f93027fcbe106cd437c2fcab34483512dd1cb45ada8192\": container with ID starting with 79dc457f2d05955902f93027fcbe106cd437c2fcab34483512dd1cb45ada8192 not found: ID does not exist" containerID="79dc457f2d05955902f93027fcbe106cd437c2fcab34483512dd1cb45ada8192" Jul 25 01:47:42.516706 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.516673 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79dc457f2d05955902f93027fcbe106cd437c2fcab34483512dd1cb45ada8192"} err="failed to get container status \"79dc457f2d05955902f93027fcbe106cd437c2fcab34483512dd1cb45ada8192\": rpc error: code = NotFound desc = could not find container \"79dc457f2d05955902f93027fcbe106cd437c2fcab34483512dd1cb45ada8192\": container with ID starting with 79dc457f2d05955902f93027fcbe106cd437c2fcab34483512dd1cb45ada8192 not found: ID does not exist" Jul 25 01:47:42.516706 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.516690 2554 scope.go:117] "RemoveContainer" containerID="cd42ba7cbf1f71b9c40c97bf1d55c986f399918237f30f6e9db917a1c11f5de5" Jul 25 01:47:42.523186 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.523163 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r"] Jul 25 01:47:42.525441 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.525425 2554 scope.go:117] "RemoveContainer" containerID="cd42ba7cbf1f71b9c40c97bf1d55c986f399918237f30f6e9db917a1c11f5de5" Jul 25 01:47:42.525691 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:47:42.525673 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd42ba7cbf1f71b9c40c97bf1d55c986f399918237f30f6e9db917a1c11f5de5\": container with ID starting with cd42ba7cbf1f71b9c40c97bf1d55c986f399918237f30f6e9db917a1c11f5de5 not found: ID does not exist" containerID="cd42ba7cbf1f71b9c40c97bf1d55c986f399918237f30f6e9db917a1c11f5de5" Jul 25 01:47:42.525744 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.525698 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd42ba7cbf1f71b9c40c97bf1d55c986f399918237f30f6e9db917a1c11f5de5"} err="failed to get container status \"cd42ba7cbf1f71b9c40c97bf1d55c986f399918237f30f6e9db917a1c11f5de5\": rpc error: code = NotFound desc = could not find container \"cd42ba7cbf1f71b9c40c97bf1d55c986f399918237f30f6e9db917a1c11f5de5\": container with ID starting with cd42ba7cbf1f71b9c40c97bf1d55c986f399918237f30f6e9db917a1c11f5de5 not found: ID does not exist" Jul 25 01:47:42.527014 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.526994 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-db7d7068-758c48f97-7mm2r"] Jul 25 01:47:42.535872 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.535849 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4"] Jul 25 01:47:42.541106 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.541084 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-db7d7068-b6548f944-brsz4"] Jul 25 01:47:42.550957 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.550927 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n"] Jul 25 01:47:42.554159 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:42.554141 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-db7d7068-699d4bd579-mrc8n"] Jul 25 01:47:43.381457 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:43.381424 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23280ea6-f43e-4736-a355-c15d0f5a0327" path="/var/lib/kubelet/pods/23280ea6-f43e-4736-a355-c15d0f5a0327/volumes" Jul 25 01:47:43.381973 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:43.381768 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="331f599b-57be-4365-af8b-51fa1653e35a" path="/var/lib/kubelet/pods/331f599b-57be-4365-af8b-51fa1653e35a/volumes" Jul 25 01:47:43.382116 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:43.382100 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="776baa6a-afd7-4f8b-9757-9bd7a6fee63b" path="/var/lib/kubelet/pods/776baa6a-afd7-4f8b-9757-9bd7a6fee63b/volumes" Jul 25 01:47:49.625666 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.625632 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5"] Jul 25 01:47:49.626161 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.626150 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="331f599b-57be-4365-af8b-51fa1653e35a" containerName="payload-processing" Jul 25 01:47:49.626222 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.626166 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="331f599b-57be-4365-af8b-51fa1653e35a" containerName="payload-processing" Jul 25 01:47:49.626222 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.626190 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776baa6a-afd7-4f8b-9757-9bd7a6fee63b" containerName="payload-pre-processing" Jul 25 01:47:49.626222 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.626199 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="776baa6a-afd7-4f8b-9757-9bd7a6fee63b" containerName="payload-pre-processing" Jul 25 01:47:49.626222 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.626213 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23280ea6-f43e-4736-a355-c15d0f5a0327" containerName="maas-api" Jul 25 01:47:49.626222 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.626221 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="23280ea6-f43e-4736-a355-c15d0f5a0327" containerName="maas-api" Jul 25 01:47:49.626463 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.626310 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="776baa6a-afd7-4f8b-9757-9bd7a6fee63b" containerName="payload-pre-processing" Jul 25 01:47:49.626463 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.626323 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="23280ea6-f43e-4736-a355-c15d0f5a0327" containerName="maas-api" Jul 25 01:47:49.626463 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.626333 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="331f599b-57be-4365-af8b-51fa1653e35a" containerName="payload-processing" Jul 25 01:47:49.631654 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.631629 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" Jul 25 01:47:49.633751 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.633727 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9f2c0ee3-serving-cert\"" Jul 25 01:47:49.638452 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.638428 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5"] Jul 25 01:47:49.752155 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.752117 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndqlz\" (UniqueName: \"kubernetes.io/projected/fae2811e-3e1f-49bc-9488-080fc52be1f3-kube-api-access-ndqlz\") pod \"maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5\" (UID: \"fae2811e-3e1f-49bc-9488-080fc52be1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" Jul 25 01:47:49.752155 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.752159 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae2811e-3e1f-49bc-9488-080fc52be1f3-openshift-service-ca\") pod \"maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5\" (UID: \"fae2811e-3e1f-49bc-9488-080fc52be1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" Jul 25 01:47:49.752425 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.752261 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fae2811e-3e1f-49bc-9488-080fc52be1f3-maas-api-tls\") pod \"maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5\" (UID: \"fae2811e-3e1f-49bc-9488-080fc52be1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" Jul 25 01:47:49.853233 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.853199 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndqlz\" (UniqueName: \"kubernetes.io/projected/fae2811e-3e1f-49bc-9488-080fc52be1f3-kube-api-access-ndqlz\") pod \"maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5\" (UID: \"fae2811e-3e1f-49bc-9488-080fc52be1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" Jul 25 01:47:49.853233 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.853238 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae2811e-3e1f-49bc-9488-080fc52be1f3-openshift-service-ca\") pod \"maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5\" (UID: \"fae2811e-3e1f-49bc-9488-080fc52be1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" Jul 25 01:47:49.853448 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.853428 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fae2811e-3e1f-49bc-9488-080fc52be1f3-maas-api-tls\") pod \"maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5\" (UID: \"fae2811e-3e1f-49bc-9488-080fc52be1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" Jul 25 01:47:49.853581 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:47:49.853563 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-serving-cert: secret "maas-api-e2e-ait-9f2c0ee3-serving-cert" not found Jul 25 01:47:49.853662 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:47:49.853637 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae2811e-3e1f-49bc-9488-080fc52be1f3-maas-api-tls podName:fae2811e-3e1f-49bc-9488-080fc52be1f3 nodeName:}" failed. No retries permitted until 2026-07-25 01:47:50.353617619 +0000 UTC m=+2211.475103455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fae2811e-3e1f-49bc-9488-080fc52be1f3-maas-api-tls") pod "maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" (UID: "fae2811e-3e1f-49bc-9488-080fc52be1f3") : secret "maas-api-e2e-ait-9f2c0ee3-serving-cert" not found Jul 25 01:47:49.853881 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.853864 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae2811e-3e1f-49bc-9488-080fc52be1f3-openshift-service-ca\") pod \"maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5\" (UID: \"fae2811e-3e1f-49bc-9488-080fc52be1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" Jul 25 01:47:49.865433 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:49.865411 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndqlz\" (UniqueName: \"kubernetes.io/projected/fae2811e-3e1f-49bc-9488-080fc52be1f3-kube-api-access-ndqlz\") pod \"maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5\" (UID: \"fae2811e-3e1f-49bc-9488-080fc52be1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" Jul 25 01:47:50.316846 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.316793 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9"] Jul 25 01:47:50.320538 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.320514 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9" Jul 25 01:47:50.323439 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.323420 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9f2c0ee3-dockercfg-l8fjb\"" Jul 25 01:47:50.323775 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.323760 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9f2c0ee3\"" Jul 25 01:47:50.334062 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.334037 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9"] Jul 25 01:47:50.349989 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.349949 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v"] Jul 25 01:47:50.353366 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.353351 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v" Jul 25 01:47:50.358904 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.358878 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fae2811e-3e1f-49bc-9488-080fc52be1f3-maas-api-tls\") pod \"maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5\" (UID: \"fae2811e-3e1f-49bc-9488-080fc52be1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" Jul 25 01:47:50.360387 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.360363 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v"] Jul 25 01:47:50.362044 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.362016 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fae2811e-3e1f-49bc-9488-080fc52be1f3-maas-api-tls\") pod \"maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5\" (UID: \"fae2811e-3e1f-49bc-9488-080fc52be1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" Jul 25 01:47:50.460386 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.460348 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31ce8b2-b9c2-4618-a672-e3b607a29d2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9\" (UID: \"b31ce8b2-b9c2-4618-a672-e3b607a29d2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9" Jul 25 01:47:50.460552 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.460401 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0e8816-cb00-4642-8f53-7504cf2774d8-plugins-config-volume\") pod \"payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v\" (UID: \"2d0e8816-cb00-4642-8f53-7504cf2774d8\") " pod="openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v" Jul 25 01:47:50.460552 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.460476 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc5z4\" (UniqueName: \"kubernetes.io/projected/b31ce8b2-b9c2-4618-a672-e3b607a29d2b-kube-api-access-kc5z4\") pod \"payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9\" (UID: \"b31ce8b2-b9c2-4618-a672-e3b607a29d2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9" Jul 25 01:47:50.460552 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.460520 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jcv\" (UniqueName: \"kubernetes.io/projected/2d0e8816-cb00-4642-8f53-7504cf2774d8-kube-api-access-s5jcv\") pod \"payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v\" (UID: \"2d0e8816-cb00-4642-8f53-7504cf2774d8\") " pod="openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v" Jul 25 01:47:50.544563 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.544531 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" Jul 25 01:47:50.561875 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.561844 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31ce8b2-b9c2-4618-a672-e3b607a29d2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9\" (UID: \"b31ce8b2-b9c2-4618-a672-e3b607a29d2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9" Jul 25 01:47:50.562003 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.561933 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0e8816-cb00-4642-8f53-7504cf2774d8-plugins-config-volume\") pod \"payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v\" (UID: \"2d0e8816-cb00-4642-8f53-7504cf2774d8\") " pod="openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v" Jul 25 01:47:50.562331 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.562306 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc5z4\" (UniqueName: \"kubernetes.io/projected/b31ce8b2-b9c2-4618-a672-e3b607a29d2b-kube-api-access-kc5z4\") pod \"payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9\" (UID: \"b31ce8b2-b9c2-4618-a672-e3b607a29d2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9" Jul 25 01:47:50.562445 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.562371 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jcv\" (UniqueName: \"kubernetes.io/projected/2d0e8816-cb00-4642-8f53-7504cf2774d8-kube-api-access-s5jcv\") pod \"payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v\" (UID: \"2d0e8816-cb00-4642-8f53-7504cf2774d8\") " pod="openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v" Jul 25 01:47:50.562823 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.562780 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31ce8b2-b9c2-4618-a672-e3b607a29d2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9\" (UID: \"b31ce8b2-b9c2-4618-a672-e3b607a29d2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9" Jul 25 01:47:50.563089 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.563065 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0e8816-cb00-4642-8f53-7504cf2774d8-plugins-config-volume\") pod \"payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v\" (UID: \"2d0e8816-cb00-4642-8f53-7504cf2774d8\") " pod="openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v" Jul 25 01:47:50.572934 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.572849 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jcv\" (UniqueName: \"kubernetes.io/projected/2d0e8816-cb00-4642-8f53-7504cf2774d8-kube-api-access-s5jcv\") pod \"payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v\" (UID: \"2d0e8816-cb00-4642-8f53-7504cf2774d8\") " pod="openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v" Jul 25 01:47:50.573092 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.572954 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc5z4\" (UniqueName: \"kubernetes.io/projected/b31ce8b2-b9c2-4618-a672-e3b607a29d2b-kube-api-access-kc5z4\") pod \"payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9\" (UID: \"b31ce8b2-b9c2-4618-a672-e3b607a29d2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9" Jul 25 01:47:50.631013 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.630977 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9" Jul 25 01:47:50.664201 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.664145 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v" Jul 25 01:47:50.694273 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.694228 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5"] Jul 25 01:47:50.695447 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:47:50.695413 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae2811e_3e1f_49bc_9488_080fc52be1f3.slice/crio-a15535836ee946ce7bb7dac5bad6b7b9f67ad22161ffa3f194cb2c6d6481f1c8 WatchSource:0}: Error finding container a15535836ee946ce7bb7dac5bad6b7b9f67ad22161ffa3f194cb2c6d6481f1c8: Status 404 returned error can't find the container with id a15535836ee946ce7bb7dac5bad6b7b9f67ad22161ffa3f194cb2c6d6481f1c8 Jul 25 01:47:50.788213 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.788183 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9"] Jul 25 01:47:50.789624 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:47:50.789594 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31ce8b2_b9c2_4618_a672_e3b607a29d2b.slice/crio-8b248f8e3f4840dbdc6149cbecdaf4f90d583984738d5d888a36aa986240fb68 WatchSource:0}: Error finding container 8b248f8e3f4840dbdc6149cbecdaf4f90d583984738d5d888a36aa986240fb68: Status 404 returned error can't find the container with id 8b248f8e3f4840dbdc6149cbecdaf4f90d583984738d5d888a36aa986240fb68 Jul 25 01:47:50.822058 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:50.822035 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v"] Jul 25 01:47:50.832865 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:47:50.832839 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0e8816_cb00_4642_8f53_7504cf2774d8.slice/crio-11117f7fb4eea2c81f8361d6bba48aecc57588ca5453ad633f3535585c68fcb2 WatchSource:0}: Error finding container 11117f7fb4eea2c81f8361d6bba48aecc57588ca5453ad633f3535585c68fcb2: Status 404 returned error can't find the container with id 11117f7fb4eea2c81f8361d6bba48aecc57588ca5453ad633f3535585c68fcb2 Jul 25 01:47:51.548224 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:51.548125 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v" event={"ID":"2d0e8816-cb00-4642-8f53-7504cf2774d8","Type":"ContainerStarted","Data":"8dd4b32a78ced965759670833062c4244f71c70150272a315e4fe49d311d32ff"} Jul 25 01:47:51.548224 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:51.548204 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v" Jul 25 01:47:51.548411 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:51.548232 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v" event={"ID":"2d0e8816-cb00-4642-8f53-7504cf2774d8","Type":"ContainerStarted","Data":"11117f7fb4eea2c81f8361d6bba48aecc57588ca5453ad633f3535585c68fcb2"} Jul 25 01:47:51.549667 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:51.549639 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" event={"ID":"fae2811e-3e1f-49bc-9488-080fc52be1f3","Type":"ContainerStarted","Data":"d917c99f34d5e23f958e21d9ccbf05b2999b731288d4e77dab5337c9cd8ce45c"} Jul 25 01:47:51.549792 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:51.549691 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" event={"ID":"fae2811e-3e1f-49bc-9488-080fc52be1f3","Type":"ContainerStarted","Data":"a15535836ee946ce7bb7dac5bad6b7b9f67ad22161ffa3f194cb2c6d6481f1c8"} Jul 25 01:47:51.549792 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:51.549722 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" Jul 25 01:47:51.551152 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:51.551127 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9" event={"ID":"b31ce8b2-b9c2-4618-a672-e3b607a29d2b","Type":"ContainerStarted","Data":"0b1b899f2010d21cb33994a37ed05e7dc2a3d6a1f88939b076c8e9453a5635c1"} Jul 25 01:47:51.551231 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:51.551168 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9" event={"ID":"b31ce8b2-b9c2-4618-a672-e3b607a29d2b","Type":"ContainerStarted","Data":"8b248f8e3f4840dbdc6149cbecdaf4f90d583984738d5d888a36aa986240fb68"} Jul 25 01:47:51.551231 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:51.551206 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9" Jul 25 01:47:51.567619 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:51.567567 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v" podStartSLOduration=1.567555441 podStartE2EDuration="1.567555441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:47:51.565427578 +0000 UTC m=+2212.686913444" watchObservedRunningTime="2026-07-25 01:47:51.567555441 +0000 UTC m=+2212.689041291" Jul 25 01:47:51.581811 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:51.581763 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9" podStartSLOduration=1.581749845 podStartE2EDuration="1.581749845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:47:51.579874292 +0000 UTC m=+2212.701360142" watchObservedRunningTime="2026-07-25 01:47:51.581749845 +0000 UTC m=+2212.703235696" Jul 25 01:47:51.598766 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:51.598723 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" podStartSLOduration=2.121325975 podStartE2EDuration="2.598707162s" totalImagesPullingTime="477.381187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:47:51.59667388 +0000 UTC m=+2212.718159729" watchObservedRunningTime="2026-07-25 01:47:51.598707162 +0000 UTC m=+2212.720193058" Jul 25 01:47:57.562494 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:47:57.562462 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" Jul 25 01:48:02.559005 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:48:02.558973 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9" Jul 25 01:48:02.559502 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:48:02.559056 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v" Jul 25 01:53:06.708334 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:06.708299 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5"] Jul 25 01:53:06.709186 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:06.709148 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" podUID="fae2811e-3e1f-49bc-9488-080fc52be1f3" containerName="maas-api" containerID="cri-o://d917c99f34d5e23f958e21d9ccbf05b2999b731288d4e77dab5337c9cd8ce45c" gracePeriod=30 Jul 25 01:53:06.761486 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:06.761453 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v"] Jul 25 01:53:06.761718 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:06.761659 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v" podUID="2d0e8816-cb00-4642-8f53-7504cf2774d8" containerName="payload-processing" containerID="cri-o://8dd4b32a78ced965759670833062c4244f71c70150272a315e4fe49d311d32ff" gracePeriod=30 Jul 25 01:53:06.770708 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:06.770677 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9"] Jul 25 01:53:06.773274 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:06.770930 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9" podUID="b31ce8b2-b9c2-4618-a672-e3b607a29d2b" containerName="payload-pre-processing" containerID="cri-o://0b1b899f2010d21cb33994a37ed05e7dc2a3d6a1f88939b076c8e9453a5635c1" gracePeriod=30 Jul 25 01:53:07.033160 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.033131 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" Jul 25 01:53:07.055932 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.055542 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v" Jul 25 01:53:07.074882 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.074857 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9" Jul 25 01:53:07.153045 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.153013 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31ce8b2-b9c2-4618-a672-e3b607a29d2b-plugins-config-volume\") pod \"b31ce8b2-b9c2-4618-a672-e3b607a29d2b\" (UID: \"b31ce8b2-b9c2-4618-a672-e3b607a29d2b\") " Jul 25 01:53:07.153212 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.153077 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndqlz\" (UniqueName: \"kubernetes.io/projected/fae2811e-3e1f-49bc-9488-080fc52be1f3-kube-api-access-ndqlz\") pod \"fae2811e-3e1f-49bc-9488-080fc52be1f3\" (UID: \"fae2811e-3e1f-49bc-9488-080fc52be1f3\") " Jul 25 01:53:07.153212 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.153124 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5jcv\" (UniqueName: \"kubernetes.io/projected/2d0e8816-cb00-4642-8f53-7504cf2774d8-kube-api-access-s5jcv\") pod \"2d0e8816-cb00-4642-8f53-7504cf2774d8\" (UID: \"2d0e8816-cb00-4642-8f53-7504cf2774d8\") " Jul 25 01:53:07.153212 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.153147 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fae2811e-3e1f-49bc-9488-080fc52be1f3-maas-api-tls\") pod \"fae2811e-3e1f-49bc-9488-080fc52be1f3\" (UID: \"fae2811e-3e1f-49bc-9488-080fc52be1f3\") " Jul 25 01:53:07.153212 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.153173 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae2811e-3e1f-49bc-9488-080fc52be1f3-openshift-service-ca\") pod \"fae2811e-3e1f-49bc-9488-080fc52be1f3\" (UID: \"fae2811e-3e1f-49bc-9488-080fc52be1f3\") " Jul 25 01:53:07.153212 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.153208 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0e8816-cb00-4642-8f53-7504cf2774d8-plugins-config-volume\") pod \"2d0e8816-cb00-4642-8f53-7504cf2774d8\" (UID: \"2d0e8816-cb00-4642-8f53-7504cf2774d8\") " Jul 25 01:53:07.153466 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.153243 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kc5z4\" (UniqueName: \"kubernetes.io/projected/b31ce8b2-b9c2-4618-a672-e3b607a29d2b-kube-api-access-kc5z4\") pod \"b31ce8b2-b9c2-4618-a672-e3b607a29d2b\" (UID: \"b31ce8b2-b9c2-4618-a672-e3b607a29d2b\") " Jul 25 01:53:07.153563 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.153531 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31ce8b2-b9c2-4618-a672-e3b607a29d2b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b31ce8b2-b9c2-4618-a672-e3b607a29d2b" (UID: "b31ce8b2-b9c2-4618-a672-e3b607a29d2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 01:53:07.153638 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.153572 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae2811e-3e1f-49bc-9488-080fc52be1f3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fae2811e-3e1f-49bc-9488-080fc52be1f3" (UID: "fae2811e-3e1f-49bc-9488-080fc52be1f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 01:53:07.153699 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.153678 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0e8816-cb00-4642-8f53-7504cf2774d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d0e8816-cb00-4642-8f53-7504cf2774d8" (UID: "2d0e8816-cb00-4642-8f53-7504cf2774d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 01:53:07.155448 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.155404 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae2811e-3e1f-49bc-9488-080fc52be1f3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fae2811e-3e1f-49bc-9488-080fc52be1f3" (UID: "fae2811e-3e1f-49bc-9488-080fc52be1f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 01:53:07.155562 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.155481 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae2811e-3e1f-49bc-9488-080fc52be1f3-kube-api-access-ndqlz" (OuterVolumeSpecName: "kube-api-access-ndqlz") pod "fae2811e-3e1f-49bc-9488-080fc52be1f3" (UID: "fae2811e-3e1f-49bc-9488-080fc52be1f3"). InnerVolumeSpecName "kube-api-access-ndqlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:53:07.155562 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.155486 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0e8816-cb00-4642-8f53-7504cf2774d8-kube-api-access-s5jcv" (OuterVolumeSpecName: "kube-api-access-s5jcv") pod "2d0e8816-cb00-4642-8f53-7504cf2774d8" (UID: "2d0e8816-cb00-4642-8f53-7504cf2774d8"). InnerVolumeSpecName "kube-api-access-s5jcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:53:07.155562 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.155555 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31ce8b2-b9c2-4618-a672-e3b607a29d2b-kube-api-access-kc5z4" (OuterVolumeSpecName: "kube-api-access-kc5z4") pod "b31ce8b2-b9c2-4618-a672-e3b607a29d2b" (UID: "b31ce8b2-b9c2-4618-a672-e3b607a29d2b"). InnerVolumeSpecName "kube-api-access-kc5z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:53:07.225974 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.225895 2554 generic.go:356] "Generic (PLEG): container finished" podID="2d0e8816-cb00-4642-8f53-7504cf2774d8" containerID="8dd4b32a78ced965759670833062c4244f71c70150272a315e4fe49d311d32ff" exitCode=0 Jul 25 01:53:07.225974 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.225966 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v" Jul 25 01:53:07.226161 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.225989 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v" event={"ID":"2d0e8816-cb00-4642-8f53-7504cf2774d8","Type":"ContainerDied","Data":"8dd4b32a78ced965759670833062c4244f71c70150272a315e4fe49d311d32ff"} Jul 25 01:53:07.226161 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.226075 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v" event={"ID":"2d0e8816-cb00-4642-8f53-7504cf2774d8","Type":"ContainerDied","Data":"11117f7fb4eea2c81f8361d6bba48aecc57588ca5453ad633f3535585c68fcb2"} Jul 25 01:53:07.226161 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.226101 2554 scope.go:117] "RemoveContainer" containerID="8dd4b32a78ced965759670833062c4244f71c70150272a315e4fe49d311d32ff" Jul 25 01:53:07.227262 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.227237 2554 generic.go:356] "Generic (PLEG): container finished" podID="fae2811e-3e1f-49bc-9488-080fc52be1f3" containerID="d917c99f34d5e23f958e21d9ccbf05b2999b731288d4e77dab5337c9cd8ce45c" exitCode=0 Jul 25 01:53:07.227363 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.227267 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" event={"ID":"fae2811e-3e1f-49bc-9488-080fc52be1f3","Type":"ContainerDied","Data":"d917c99f34d5e23f958e21d9ccbf05b2999b731288d4e77dab5337c9cd8ce45c"} Jul 25 01:53:07.227363 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.227286 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" Jul 25 01:53:07.227363 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.227313 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5" event={"ID":"fae2811e-3e1f-49bc-9488-080fc52be1f3","Type":"ContainerDied","Data":"a15535836ee946ce7bb7dac5bad6b7b9f67ad22161ffa3f194cb2c6d6481f1c8"} Jul 25 01:53:07.228856 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.228837 2554 generic.go:356] "Generic (PLEG): container finished" podID="b31ce8b2-b9c2-4618-a672-e3b607a29d2b" containerID="0b1b899f2010d21cb33994a37ed05e7dc2a3d6a1f88939b076c8e9453a5635c1" exitCode=0 Jul 25 01:53:07.228958 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.228863 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9" event={"ID":"b31ce8b2-b9c2-4618-a672-e3b607a29d2b","Type":"ContainerDied","Data":"0b1b899f2010d21cb33994a37ed05e7dc2a3d6a1f88939b076c8e9453a5635c1"} Jul 25 01:53:07.228958 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.228883 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9" Jul 25 01:53:07.228958 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.228913 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9" event={"ID":"b31ce8b2-b9c2-4618-a672-e3b607a29d2b","Type":"ContainerDied","Data":"8b248f8e3f4840dbdc6149cbecdaf4f90d583984738d5d888a36aa986240fb68"} Jul 25 01:53:07.237666 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.237645 2554 scope.go:117] "RemoveContainer" containerID="8dd4b32a78ced965759670833062c4244f71c70150272a315e4fe49d311d32ff" Jul 25 01:53:07.238061 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:53:07.237936 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd4b32a78ced965759670833062c4244f71c70150272a315e4fe49d311d32ff\": container with ID starting with 8dd4b32a78ced965759670833062c4244f71c70150272a315e4fe49d311d32ff not found: ID does not exist" containerID="8dd4b32a78ced965759670833062c4244f71c70150272a315e4fe49d311d32ff" Jul 25 01:53:07.238061 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.237961 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd4b32a78ced965759670833062c4244f71c70150272a315e4fe49d311d32ff"} err="failed to get container status \"8dd4b32a78ced965759670833062c4244f71c70150272a315e4fe49d311d32ff\": rpc error: code = NotFound desc = could not find container \"8dd4b32a78ced965759670833062c4244f71c70150272a315e4fe49d311d32ff\": container with ID starting with 8dd4b32a78ced965759670833062c4244f71c70150272a315e4fe49d311d32ff not found: ID does not exist" Jul 25 01:53:07.238061 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.237977 2554 scope.go:117] "RemoveContainer" containerID="d917c99f34d5e23f958e21d9ccbf05b2999b731288d4e77dab5337c9cd8ce45c" Jul 25 01:53:07.246727 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.246712 2554 scope.go:117] "RemoveContainer" containerID="d917c99f34d5e23f958e21d9ccbf05b2999b731288d4e77dab5337c9cd8ce45c" Jul 25 01:53:07.246965 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:53:07.246947 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d917c99f34d5e23f958e21d9ccbf05b2999b731288d4e77dab5337c9cd8ce45c\": container with ID starting with d917c99f34d5e23f958e21d9ccbf05b2999b731288d4e77dab5337c9cd8ce45c not found: ID does not exist" containerID="d917c99f34d5e23f958e21d9ccbf05b2999b731288d4e77dab5337c9cd8ce45c" Jul 25 01:53:07.247046 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.246976 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d917c99f34d5e23f958e21d9ccbf05b2999b731288d4e77dab5337c9cd8ce45c"} err="failed to get container status \"d917c99f34d5e23f958e21d9ccbf05b2999b731288d4e77dab5337c9cd8ce45c\": rpc error: code = NotFound desc = could not find container \"d917c99f34d5e23f958e21d9ccbf05b2999b731288d4e77dab5337c9cd8ce45c\": container with ID starting with d917c99f34d5e23f958e21d9ccbf05b2999b731288d4e77dab5337c9cd8ce45c not found: ID does not exist" Jul 25 01:53:07.247046 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.246998 2554 scope.go:117] "RemoveContainer" containerID="0b1b899f2010d21cb33994a37ed05e7dc2a3d6a1f88939b076c8e9453a5635c1" Jul 25 01:53:07.251925 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.251907 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5"] Jul 25 01:53:07.254149 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.254128 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9f2c0ee3-57d9846687-q98b5"] Jul 25 01:53:07.254237 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.254167 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc5z4\" (UniqueName: \"kubernetes.io/projected/b31ce8b2-b9c2-4618-a672-e3b607a29d2b-kube-api-access-kc5z4\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:53:07.254237 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.254188 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31ce8b2-b9c2-4618-a672-e3b607a29d2b-plugins-config-volume\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:53:07.254237 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.254199 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndqlz\" (UniqueName: \"kubernetes.io/projected/fae2811e-3e1f-49bc-9488-080fc52be1f3-kube-api-access-ndqlz\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:53:07.254237 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.254207 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5jcv\" (UniqueName: \"kubernetes.io/projected/2d0e8816-cb00-4642-8f53-7504cf2774d8-kube-api-access-s5jcv\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:53:07.254237 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.254217 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fae2811e-3e1f-49bc-9488-080fc52be1f3-maas-api-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:53:07.254237 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.254225 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae2811e-3e1f-49bc-9488-080fc52be1f3-openshift-service-ca\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:53:07.254237 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.254234 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0e8816-cb00-4642-8f53-7504cf2774d8-plugins-config-volume\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:53:07.255415 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.255397 2554 scope.go:117] "RemoveContainer" containerID="0b1b899f2010d21cb33994a37ed05e7dc2a3d6a1f88939b076c8e9453a5635c1" Jul 25 01:53:07.255671 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:53:07.255651 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b1b899f2010d21cb33994a37ed05e7dc2a3d6a1f88939b076c8e9453a5635c1\": container with ID starting with 0b1b899f2010d21cb33994a37ed05e7dc2a3d6a1f88939b076c8e9453a5635c1 not found: ID does not exist" containerID="0b1b899f2010d21cb33994a37ed05e7dc2a3d6a1f88939b076c8e9453a5635c1" Jul 25 01:53:07.255726 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.255676 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b1b899f2010d21cb33994a37ed05e7dc2a3d6a1f88939b076c8e9453a5635c1"} err="failed to get container status \"0b1b899f2010d21cb33994a37ed05e7dc2a3d6a1f88939b076c8e9453a5635c1\": rpc error: code = NotFound desc = could not find container \"0b1b899f2010d21cb33994a37ed05e7dc2a3d6a1f88939b076c8e9453a5635c1\": container with ID starting with 0b1b899f2010d21cb33994a37ed05e7dc2a3d6a1f88939b076c8e9453a5635c1 not found: ID does not exist" Jul 25 01:53:07.262022 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.262000 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v"] Jul 25 01:53:07.265068 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.265040 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9f2c0ee3-854b64bcfd-mk29v"] Jul 25 01:53:07.275521 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.275500 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9"] Jul 25 01:53:07.279153 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.279132 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9f2c0ee3-f59ffc844-fk7l9"] Jul 25 01:53:07.381030 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.380998 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d0e8816-cb00-4642-8f53-7504cf2774d8" path="/var/lib/kubelet/pods/2d0e8816-cb00-4642-8f53-7504cf2774d8/volumes" Jul 25 01:53:07.381359 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.381346 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b31ce8b2-b9c2-4618-a672-e3b607a29d2b" path="/var/lib/kubelet/pods/b31ce8b2-b9c2-4618-a672-e3b607a29d2b/volumes" Jul 25 01:53:07.381693 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:07.381680 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fae2811e-3e1f-49bc-9488-080fc52be1f3" path="/var/lib/kubelet/pods/fae2811e-3e1f-49bc-9488-080fc52be1f3/volumes" Jul 25 01:53:14.376082 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.376036 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45"] Jul 25 01:53:14.377227 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.377196 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b31ce8b2-b9c2-4618-a672-e3b607a29d2b" containerName="payload-pre-processing" Jul 25 01:53:14.377227 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.377223 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b31ce8b2-b9c2-4618-a672-e3b607a29d2b" containerName="payload-pre-processing" Jul 25 01:53:14.377430 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.377242 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d0e8816-cb00-4642-8f53-7504cf2774d8" containerName="payload-processing" Jul 25 01:53:14.377430 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.377251 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d0e8816-cb00-4642-8f53-7504cf2774d8" containerName="payload-processing" Jul 25 01:53:14.377430 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.377282 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae2811e-3e1f-49bc-9488-080fc52be1f3" containerName="maas-api" Jul 25 01:53:14.377430 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.377291 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae2811e-3e1f-49bc-9488-080fc52be1f3" containerName="maas-api" Jul 25 01:53:14.377430 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.377397 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fae2811e-3e1f-49bc-9488-080fc52be1f3" containerName="maas-api" Jul 25 01:53:14.377430 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.377410 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b31ce8b2-b9c2-4618-a672-e3b607a29d2b" containerName="payload-pre-processing" Jul 25 01:53:14.377430 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.377419 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d0e8816-cb00-4642-8f53-7504cf2774d8" containerName="payload-processing" Jul 25 01:53:14.382554 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.382532 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" Jul 25 01:53:14.384558 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.384537 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ed0d590f-serving-cert\"" Jul 25 01:53:14.392216 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.392193 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45"] Jul 25 01:53:14.516648 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.516606 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2301279c-4b5f-42f6-8532-00547d6e4b80-maas-api-tls\") pod \"maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45\" (UID: \"2301279c-4b5f-42f6-8532-00547d6e4b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" Jul 25 01:53:14.516823 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.516652 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2301279c-4b5f-42f6-8532-00547d6e4b80-openshift-service-ca\") pod \"maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45\" (UID: \"2301279c-4b5f-42f6-8532-00547d6e4b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" Jul 25 01:53:14.516823 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.516678 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n9jh\" (UniqueName: \"kubernetes.io/projected/2301279c-4b5f-42f6-8532-00547d6e4b80-kube-api-access-5n9jh\") pod \"maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45\" (UID: \"2301279c-4b5f-42f6-8532-00547d6e4b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" Jul 25 01:53:14.617343 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.617310 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2301279c-4b5f-42f6-8532-00547d6e4b80-maas-api-tls\") pod \"maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45\" (UID: \"2301279c-4b5f-42f6-8532-00547d6e4b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" Jul 25 01:53:14.617522 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.617350 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2301279c-4b5f-42f6-8532-00547d6e4b80-openshift-service-ca\") pod \"maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45\" (UID: \"2301279c-4b5f-42f6-8532-00547d6e4b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" Jul 25 01:53:14.617522 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.617368 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n9jh\" (UniqueName: \"kubernetes.io/projected/2301279c-4b5f-42f6-8532-00547d6e4b80-kube-api-access-5n9jh\") pod \"maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45\" (UID: \"2301279c-4b5f-42f6-8532-00547d6e4b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" Jul 25 01:53:14.617522 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:53:14.617464 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-serving-cert: secret "maas-api-e2e-derive-ed0d590f-serving-cert" not found Jul 25 01:53:14.617702 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:53:14.617542 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2301279c-4b5f-42f6-8532-00547d6e4b80-maas-api-tls podName:2301279c-4b5f-42f6-8532-00547d6e4b80 nodeName:}" failed. No retries permitted until 2026-07-25 01:53:15.117523952 +0000 UTC m=+2536.239009779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2301279c-4b5f-42f6-8532-00547d6e4b80-maas-api-tls") pod "maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" (UID: "2301279c-4b5f-42f6-8532-00547d6e4b80") : secret "maas-api-e2e-derive-ed0d590f-serving-cert" not found Jul 25 01:53:14.618078 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.618060 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2301279c-4b5f-42f6-8532-00547d6e4b80-openshift-service-ca\") pod \"maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45\" (UID: \"2301279c-4b5f-42f6-8532-00547d6e4b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" Jul 25 01:53:14.627069 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:14.627012 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n9jh\" (UniqueName: \"kubernetes.io/projected/2301279c-4b5f-42f6-8532-00547d6e4b80-kube-api-access-5n9jh\") pod \"maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45\" (UID: \"2301279c-4b5f-42f6-8532-00547d6e4b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" Jul 25 01:53:15.012752 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:15.012710 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr"] Jul 25 01:53:15.016388 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:15.016364 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr" Jul 25 01:53:15.018594 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:15.018571 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ed0d590f-dockercfg-hhtn7\"" Jul 25 01:53:15.018697 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:15.018604 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ed0d590f\"" Jul 25 01:53:15.024119 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:15.024096 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr"] Jul 25 01:53:15.123055 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:15.123019 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk88b\" (UniqueName: \"kubernetes.io/projected/7a852603-b432-4277-9fd1-a115982c62a3-kube-api-access-nk88b\") pod \"payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr\" (UID: \"7a852603-b432-4277-9fd1-a115982c62a3\") " pod="openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr" Jul 25 01:53:15.123219 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:15.123085 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2301279c-4b5f-42f6-8532-00547d6e4b80-maas-api-tls\") pod \"maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45\" (UID: \"2301279c-4b5f-42f6-8532-00547d6e4b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" Jul 25 01:53:15.123270 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:15.123217 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a852603-b432-4277-9fd1-a115982c62a3-plugins-config-volume\") pod \"payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr\" (UID: \"7a852603-b432-4277-9fd1-a115982c62a3\") " pod="openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr" Jul 25 01:53:15.125674 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:15.125651 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2301279c-4b5f-42f6-8532-00547d6e4b80-maas-api-tls\") pod \"maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45\" (UID: \"2301279c-4b5f-42f6-8532-00547d6e4b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" Jul 25 01:53:15.224734 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:15.224703 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk88b\" (UniqueName: \"kubernetes.io/projected/7a852603-b432-4277-9fd1-a115982c62a3-kube-api-access-nk88b\") pod \"payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr\" (UID: \"7a852603-b432-4277-9fd1-a115982c62a3\") " pod="openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr" Jul 25 01:53:15.224916 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:15.224826 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a852603-b432-4277-9fd1-a115982c62a3-plugins-config-volume\") pod \"payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr\" (UID: \"7a852603-b432-4277-9fd1-a115982c62a3\") " pod="openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr" Jul 25 01:53:15.225430 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:15.225412 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a852603-b432-4277-9fd1-a115982c62a3-plugins-config-volume\") pod \"payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr\" (UID: \"7a852603-b432-4277-9fd1-a115982c62a3\") " pod="openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr" Jul 25 01:53:15.233318 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:15.233286 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk88b\" (UniqueName: \"kubernetes.io/projected/7a852603-b432-4277-9fd1-a115982c62a3-kube-api-access-nk88b\") pod \"payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr\" (UID: \"7a852603-b432-4277-9fd1-a115982c62a3\") " pod="openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr" Jul 25 01:53:15.296287 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:15.296205 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" Jul 25 01:53:15.330419 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:15.330385 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr" Jul 25 01:53:15.448267 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:15.448174 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45"] Jul 25 01:53:15.452764 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:53:15.452730 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2301279c_4b5f_42f6_8532_00547d6e4b80.slice/crio-3d6560c3ee0b6ab55b759d7a4cf6b6a6ffb450182af8578bf6c69b5faa26963b WatchSource:0}: Error finding container 3d6560c3ee0b6ab55b759d7a4cf6b6a6ffb450182af8578bf6c69b5faa26963b: Status 404 returned error can't find the container with id 3d6560c3ee0b6ab55b759d7a4cf6b6a6ffb450182af8578bf6c69b5faa26963b Jul 25 01:53:15.454293 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:15.454274 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 25 01:53:15.690540 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:15.690396 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr"] Jul 25 01:53:15.693397 ip-10-0-135-43 kubenswrapper[2554]: W0725 01:53:15.693370 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a852603_b432_4277_9fd1_a115982c62a3.slice/crio-3fe7d59b80371c3385f9193dd1c8ed3dff7a92a7b4a17a77c20ae8168846ad51 WatchSource:0}: Error finding container 3fe7d59b80371c3385f9193dd1c8ed3dff7a92a7b4a17a77c20ae8168846ad51: Status 404 returned error can't find the container with id 3fe7d59b80371c3385f9193dd1c8ed3dff7a92a7b4a17a77c20ae8168846ad51 Jul 25 01:53:16.285283 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:16.285201 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr" event={"ID":"7a852603-b432-4277-9fd1-a115982c62a3","Type":"ContainerStarted","Data":"3f4a8f3181acf0436ec60d89c2c28331830c67719bf50e9216284e4b39ee9133"} Jul 25 01:53:16.285283 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:16.285258 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr" event={"ID":"7a852603-b432-4277-9fd1-a115982c62a3","Type":"ContainerStarted","Data":"3fe7d59b80371c3385f9193dd1c8ed3dff7a92a7b4a17a77c20ae8168846ad51"} Jul 25 01:53:16.285501 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:16.285288 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr" Jul 25 01:53:16.286898 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:16.286858 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" event={"ID":"2301279c-4b5f-42f6-8532-00547d6e4b80","Type":"ContainerStarted","Data":"79cf27631bb3448e9845e601fc990fc7e1cd4dda20851ca647dead2b1b0fb56a"} Jul 25 01:53:16.287027 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:16.286915 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" event={"ID":"2301279c-4b5f-42f6-8532-00547d6e4b80","Type":"ContainerStarted","Data":"3d6560c3ee0b6ab55b759d7a4cf6b6a6ffb450182af8578bf6c69b5faa26963b"} Jul 25 01:53:16.287027 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:16.286969 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" Jul 25 01:53:16.301778 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:16.301725 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr" podStartSLOduration=2.301708985 podStartE2EDuration="2.301708985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:53:16.301036662 +0000 UTC m=+2537.422522512" watchObservedRunningTime="2026-07-25 01:53:16.301708985 +0000 UTC m=+2537.423194837" Jul 25 01:53:16.318081 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:16.318035 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" podStartSLOduration=1.880396426 podStartE2EDuration="2.31801766s" totalImagesPullingTime="437.621234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:53:16.316170329 +0000 UTC m=+2537.437656179" watchObservedRunningTime="2026-07-25 01:53:16.31801766 +0000 UTC m=+2537.439503512" Jul 25 01:53:22.297406 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:22.297372 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" Jul 25 01:53:27.293972 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:53:27.293939 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr" Jul 25 01:58:31.113843 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.113730 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45"] Jul 25 01:58:31.114303 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.114015 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" podUID="2301279c-4b5f-42f6-8532-00547d6e4b80" containerName="maas-api" containerID="cri-o://79cf27631bb3448e9845e601fc990fc7e1cd4dda20851ca647dead2b1b0fb56a" gracePeriod=30 Jul 25 01:58:31.161677 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.161639 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr"] Jul 25 01:58:31.162048 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.161929 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr" podUID="7a852603-b432-4277-9fd1-a115982c62a3" containerName="payload-processing" containerID="cri-o://3f4a8f3181acf0436ec60d89c2c28331830c67719bf50e9216284e4b39ee9133" gracePeriod=30 Jul 25 01:58:31.411021 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.410998 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" Jul 25 01:58:31.441107 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.441086 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr" Jul 25 01:58:31.502978 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.502945 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a852603-b432-4277-9fd1-a115982c62a3-plugins-config-volume\") pod \"7a852603-b432-4277-9fd1-a115982c62a3\" (UID: \"7a852603-b432-4277-9fd1-a115982c62a3\") " Jul 25 01:58:31.503115 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.503060 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2301279c-4b5f-42f6-8532-00547d6e4b80-maas-api-tls\") pod \"2301279c-4b5f-42f6-8532-00547d6e4b80\" (UID: \"2301279c-4b5f-42f6-8532-00547d6e4b80\") " Jul 25 01:58:31.503115 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.503106 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nk88b\" (UniqueName: \"kubernetes.io/projected/7a852603-b432-4277-9fd1-a115982c62a3-kube-api-access-nk88b\") pod \"7a852603-b432-4277-9fd1-a115982c62a3\" (UID: \"7a852603-b432-4277-9fd1-a115982c62a3\") " Jul 25 01:58:31.503192 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.503141 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n9jh\" (UniqueName: \"kubernetes.io/projected/2301279c-4b5f-42f6-8532-00547d6e4b80-kube-api-access-5n9jh\") pod \"2301279c-4b5f-42f6-8532-00547d6e4b80\" (UID: \"2301279c-4b5f-42f6-8532-00547d6e4b80\") " Jul 25 01:58:31.503192 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.503170 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2301279c-4b5f-42f6-8532-00547d6e4b80-openshift-service-ca\") pod \"2301279c-4b5f-42f6-8532-00547d6e4b80\" (UID: \"2301279c-4b5f-42f6-8532-00547d6e4b80\") " Jul 25 01:58:31.503403 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.503382 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a852603-b432-4277-9fd1-a115982c62a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7a852603-b432-4277-9fd1-a115982c62a3" (UID: "7a852603-b432-4277-9fd1-a115982c62a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 01:58:31.503626 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.503606 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a852603-b432-4277-9fd1-a115982c62a3-plugins-config-volume\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:58:31.503699 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.503619 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2301279c-4b5f-42f6-8532-00547d6e4b80-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2301279c-4b5f-42f6-8532-00547d6e4b80" (UID: "2301279c-4b5f-42f6-8532-00547d6e4b80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 01:58:31.505400 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.505380 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2301279c-4b5f-42f6-8532-00547d6e4b80-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2301279c-4b5f-42f6-8532-00547d6e4b80" (UID: "2301279c-4b5f-42f6-8532-00547d6e4b80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 01:58:31.505464 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.505413 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2301279c-4b5f-42f6-8532-00547d6e4b80-kube-api-access-5n9jh" (OuterVolumeSpecName: "kube-api-access-5n9jh") pod "2301279c-4b5f-42f6-8532-00547d6e4b80" (UID: "2301279c-4b5f-42f6-8532-00547d6e4b80"). InnerVolumeSpecName "kube-api-access-5n9jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:58:31.505464 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.505443 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a852603-b432-4277-9fd1-a115982c62a3-kube-api-access-nk88b" (OuterVolumeSpecName: "kube-api-access-nk88b") pod "7a852603-b432-4277-9fd1-a115982c62a3" (UID: "7a852603-b432-4277-9fd1-a115982c62a3"). InnerVolumeSpecName "kube-api-access-nk88b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 01:58:31.604716 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.604685 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2301279c-4b5f-42f6-8532-00547d6e4b80-maas-api-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:58:31.604716 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.604710 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk88b\" (UniqueName: \"kubernetes.io/projected/7a852603-b432-4277-9fd1-a115982c62a3-kube-api-access-nk88b\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:58:31.604716 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.604720 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n9jh\" (UniqueName: \"kubernetes.io/projected/2301279c-4b5f-42f6-8532-00547d6e4b80-kube-api-access-5n9jh\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:58:31.604936 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.604729 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2301279c-4b5f-42f6-8532-00547d6e4b80-openshift-service-ca\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 01:58:31.935972 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.935934 2554 generic.go:356] "Generic (PLEG): container finished" podID="2301279c-4b5f-42f6-8532-00547d6e4b80" containerID="79cf27631bb3448e9845e601fc990fc7e1cd4dda20851ca647dead2b1b0fb56a" exitCode=0 Jul 25 01:58:31.936136 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.935980 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" Jul 25 01:58:31.936136 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.936025 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" event={"ID":"2301279c-4b5f-42f6-8532-00547d6e4b80","Type":"ContainerDied","Data":"79cf27631bb3448e9845e601fc990fc7e1cd4dda20851ca647dead2b1b0fb56a"} Jul 25 01:58:31.936136 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.936085 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45" event={"ID":"2301279c-4b5f-42f6-8532-00547d6e4b80","Type":"ContainerDied","Data":"3d6560c3ee0b6ab55b759d7a4cf6b6a6ffb450182af8578bf6c69b5faa26963b"} Jul 25 01:58:31.936136 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.936114 2554 scope.go:117] "RemoveContainer" containerID="79cf27631bb3448e9845e601fc990fc7e1cd4dda20851ca647dead2b1b0fb56a" Jul 25 01:58:31.937552 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.937526 2554 generic.go:356] "Generic (PLEG): container finished" podID="7a852603-b432-4277-9fd1-a115982c62a3" containerID="3f4a8f3181acf0436ec60d89c2c28331830c67719bf50e9216284e4b39ee9133" exitCode=0 Jul 25 01:58:31.937657 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.937574 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr" Jul 25 01:58:31.937657 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.937581 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr" event={"ID":"7a852603-b432-4277-9fd1-a115982c62a3","Type":"ContainerDied","Data":"3f4a8f3181acf0436ec60d89c2c28331830c67719bf50e9216284e4b39ee9133"} Jul 25 01:58:31.937657 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.937621 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr" event={"ID":"7a852603-b432-4277-9fd1-a115982c62a3","Type":"ContainerDied","Data":"3fe7d59b80371c3385f9193dd1c8ed3dff7a92a7b4a17a77c20ae8168846ad51"} Jul 25 01:58:31.945789 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.945771 2554 scope.go:117] "RemoveContainer" containerID="79cf27631bb3448e9845e601fc990fc7e1cd4dda20851ca647dead2b1b0fb56a" Jul 25 01:58:31.946120 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:58:31.946093 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79cf27631bb3448e9845e601fc990fc7e1cd4dda20851ca647dead2b1b0fb56a\": container with ID starting with 79cf27631bb3448e9845e601fc990fc7e1cd4dda20851ca647dead2b1b0fb56a not found: ID does not exist" containerID="79cf27631bb3448e9845e601fc990fc7e1cd4dda20851ca647dead2b1b0fb56a" Jul 25 01:58:31.946234 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.946130 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79cf27631bb3448e9845e601fc990fc7e1cd4dda20851ca647dead2b1b0fb56a"} err="failed to get container status \"79cf27631bb3448e9845e601fc990fc7e1cd4dda20851ca647dead2b1b0fb56a\": rpc error: code = NotFound desc = could not find container \"79cf27631bb3448e9845e601fc990fc7e1cd4dda20851ca647dead2b1b0fb56a\": container with ID starting with 79cf27631bb3448e9845e601fc990fc7e1cd4dda20851ca647dead2b1b0fb56a not found: ID does not exist" Jul 25 01:58:31.946234 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.946153 2554 scope.go:117] "RemoveContainer" containerID="3f4a8f3181acf0436ec60d89c2c28331830c67719bf50e9216284e4b39ee9133" Jul 25 01:58:31.954642 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.954624 2554 scope.go:117] "RemoveContainer" containerID="3f4a8f3181acf0436ec60d89c2c28331830c67719bf50e9216284e4b39ee9133" Jul 25 01:58:31.954892 ip-10-0-135-43 kubenswrapper[2554]: E0725 01:58:31.954872 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f4a8f3181acf0436ec60d89c2c28331830c67719bf50e9216284e4b39ee9133\": container with ID starting with 3f4a8f3181acf0436ec60d89c2c28331830c67719bf50e9216284e4b39ee9133 not found: ID does not exist" containerID="3f4a8f3181acf0436ec60d89c2c28331830c67719bf50e9216284e4b39ee9133" Jul 25 01:58:31.954948 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.954899 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f4a8f3181acf0436ec60d89c2c28331830c67719bf50e9216284e4b39ee9133"} err="failed to get container status \"3f4a8f3181acf0436ec60d89c2c28331830c67719bf50e9216284e4b39ee9133\": rpc error: code = NotFound desc = could not find container \"3f4a8f3181acf0436ec60d89c2c28331830c67719bf50e9216284e4b39ee9133\": container with ID starting with 3f4a8f3181acf0436ec60d89c2c28331830c67719bf50e9216284e4b39ee9133 not found: ID does not exist" Jul 25 01:58:31.960385 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.960364 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45"] Jul 25 01:58:31.968217 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.968189 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ed0d590f-747c9456b4-l7n45"] Jul 25 01:58:31.976087 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.976063 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr"] Jul 25 01:58:31.981745 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:31.981725 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ed0d590f-9568d777d-dtfvr"] Jul 25 01:58:33.381147 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:33.381119 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2301279c-4b5f-42f6-8532-00547d6e4b80" path="/var/lib/kubelet/pods/2301279c-4b5f-42f6-8532-00547d6e4b80/volumes" Jul 25 01:58:33.381524 ip-10-0-135-43 kubenswrapper[2554]: I0725 01:58:33.381472 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a852603-b432-4277-9fd1-a115982c62a3" path="/var/lib/kubelet/pods/7a852603-b432-4277-9fd1-a115982c62a3/volumes" Jul 25 02:00:16.354233 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.354200 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk"] Jul 25 02:00:16.354781 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.354557 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a852603-b432-4277-9fd1-a115982c62a3" containerName="payload-processing" Jul 25 02:00:16.354781 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.354568 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a852603-b432-4277-9fd1-a115982c62a3" containerName="payload-processing" Jul 25 02:00:16.354781 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.354580 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2301279c-4b5f-42f6-8532-00547d6e4b80" containerName="maas-api" Jul 25 02:00:16.354781 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.354588 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2301279c-4b5f-42f6-8532-00547d6e4b80" containerName="maas-api" Jul 25 02:00:16.354781 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.354666 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a852603-b432-4277-9fd1-a115982c62a3" containerName="payload-processing" Jul 25 02:00:16.354781 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.354678 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2301279c-4b5f-42f6-8532-00547d6e4b80" containerName="maas-api" Jul 25 02:00:16.357961 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.357938 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" Jul 25 02:00:16.360042 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.360016 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-17288039-serving-cert\"" Jul 25 02:00:16.370056 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.370031 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk"] Jul 25 02:00:16.488023 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.487991 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w77wg\" (UniqueName: \"kubernetes.io/projected/06812c3e-c7f7-4792-a0a4-0b44d7583334-kube-api-access-w77wg\") pod \"maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk\" (UID: \"06812c3e-c7f7-4792-a0a4-0b44d7583334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" Jul 25 02:00:16.488217 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.488159 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06812c3e-c7f7-4792-a0a4-0b44d7583334-maas-api-tls\") pod \"maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk\" (UID: \"06812c3e-c7f7-4792-a0a4-0b44d7583334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" Jul 25 02:00:16.488217 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.488200 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06812c3e-c7f7-4792-a0a4-0b44d7583334-openshift-service-ca\") pod \"maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk\" (UID: \"06812c3e-c7f7-4792-a0a4-0b44d7583334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" Jul 25 02:00:16.589686 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.589658 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06812c3e-c7f7-4792-a0a4-0b44d7583334-maas-api-tls\") pod \"maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk\" (UID: \"06812c3e-c7f7-4792-a0a4-0b44d7583334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" Jul 25 02:00:16.589686 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.589691 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06812c3e-c7f7-4792-a0a4-0b44d7583334-openshift-service-ca\") pod \"maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk\" (UID: \"06812c3e-c7f7-4792-a0a4-0b44d7583334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" Jul 25 02:00:16.589932 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.589757 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w77wg\" (UniqueName: \"kubernetes.io/projected/06812c3e-c7f7-4792-a0a4-0b44d7583334-kube-api-access-w77wg\") pod \"maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk\" (UID: \"06812c3e-c7f7-4792-a0a4-0b44d7583334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" Jul 25 02:00:16.590491 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.590465 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06812c3e-c7f7-4792-a0a4-0b44d7583334-openshift-service-ca\") pod \"maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk\" (UID: \"06812c3e-c7f7-4792-a0a4-0b44d7583334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" Jul 25 02:00:16.592353 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.592334 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06812c3e-c7f7-4792-a0a4-0b44d7583334-maas-api-tls\") pod \"maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk\" (UID: \"06812c3e-c7f7-4792-a0a4-0b44d7583334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" Jul 25 02:00:16.597235 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.597217 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w77wg\" (UniqueName: \"kubernetes.io/projected/06812c3e-c7f7-4792-a0a4-0b44d7583334-kube-api-access-w77wg\") pod \"maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk\" (UID: \"06812c3e-c7f7-4792-a0a4-0b44d7583334\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" Jul 25 02:00:16.670928 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.670894 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" Jul 25 02:00:16.809925 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.809902 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk"] Jul 25 02:00:16.811879 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:00:16.811850 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06812c3e_c7f7_4792_a0a4_0b44d7583334.slice/crio-6b2e31f3c86ebe478b7df44fd0adfdda60807cd332560548d3e1648194d95f2c WatchSource:0}: Error finding container 6b2e31f3c86ebe478b7df44fd0adfdda60807cd332560548d3e1648194d95f2c: Status 404 returned error can't find the container with id 6b2e31f3c86ebe478b7df44fd0adfdda60807cd332560548d3e1648194d95f2c Jul 25 02:00:16.813293 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:16.813270 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 25 02:00:17.041694 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:17.041617 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8"] Jul 25 02:00:17.048381 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:17.048349 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8" Jul 25 02:00:17.051187 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:17.051164 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-17288039\"" Jul 25 02:00:17.051388 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:17.051365 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-17288039-dockercfg-hmj92\"" Jul 25 02:00:17.060778 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:17.060751 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8"] Jul 25 02:00:17.095037 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:17.095010 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62cb034-3f64-4319-b54e-ef64b89e678b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8\" (UID: \"d62cb034-3f64-4319-b54e-ef64b89e678b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8" Jul 25 02:00:17.095183 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:17.095042 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzzz7\" (UniqueName: \"kubernetes.io/projected/d62cb034-3f64-4319-b54e-ef64b89e678b-kube-api-access-kzzz7\") pod \"payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8\" (UID: \"d62cb034-3f64-4319-b54e-ef64b89e678b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8" Jul 25 02:00:17.195445 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:17.195400 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62cb034-3f64-4319-b54e-ef64b89e678b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8\" (UID: \"d62cb034-3f64-4319-b54e-ef64b89e678b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8" Jul 25 02:00:17.195445 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:17.195445 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzzz7\" (UniqueName: \"kubernetes.io/projected/d62cb034-3f64-4319-b54e-ef64b89e678b-kube-api-access-kzzz7\") pod \"payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8\" (UID: \"d62cb034-3f64-4319-b54e-ef64b89e678b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8" Jul 25 02:00:17.196119 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:17.196099 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62cb034-3f64-4319-b54e-ef64b89e678b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8\" (UID: \"d62cb034-3f64-4319-b54e-ef64b89e678b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8" Jul 25 02:00:17.203462 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:17.203437 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzzz7\" (UniqueName: \"kubernetes.io/projected/d62cb034-3f64-4319-b54e-ef64b89e678b-kube-api-access-kzzz7\") pod \"payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8\" (UID: \"d62cb034-3f64-4319-b54e-ef64b89e678b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8" Jul 25 02:00:17.360039 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:17.360014 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8" Jul 25 02:00:17.486297 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:17.486261 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" event={"ID":"06812c3e-c7f7-4792-a0a4-0b44d7583334","Type":"ContainerStarted","Data":"97da332da80a6d06d7fc4a50661a2ac8cf5d3c04e293541f21d7f47e271ce0c6"} Jul 25 02:00:17.486481 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:17.486322 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" event={"ID":"06812c3e-c7f7-4792-a0a4-0b44d7583334","Type":"ContainerStarted","Data":"6b2e31f3c86ebe478b7df44fd0adfdda60807cd332560548d3e1648194d95f2c"} Jul 25 02:00:17.486481 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:17.486379 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" Jul 25 02:00:17.500442 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:17.500411 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8"] Jul 25 02:00:17.502113 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:00:17.502092 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62cb034_3f64_4319_b54e_ef64b89e678b.slice/crio-f5f1e2dacbe0bf4412a66ef0adb8a7d51a8709297aee81fc9de163f13c9839d6 WatchSource:0}: Error finding container f5f1e2dacbe0bf4412a66ef0adb8a7d51a8709297aee81fc9de163f13c9839d6: Status 404 returned error can't find the container with id f5f1e2dacbe0bf4412a66ef0adb8a7d51a8709297aee81fc9de163f13c9839d6 Jul 25 02:00:17.507032 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:17.506975 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" podStartSLOduration=1.016998004 podStartE2EDuration="1.506960239s" totalImagesPullingTime="489.962235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:00:17.504575793 +0000 UTC m=+2958.626061644" watchObservedRunningTime="2026-07-25 02:00:17.506960239 +0000 UTC m=+2958.628446133" Jul 25 02:00:18.494666 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:18.494633 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8" event={"ID":"d62cb034-3f64-4319-b54e-ef64b89e678b","Type":"ContainerStarted","Data":"771ed58ffd534bb563176c35465d19f51bb2cee13c760ce68c2380621e7b0e8c"} Jul 25 02:00:18.495102 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:18.494690 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8" event={"ID":"d62cb034-3f64-4319-b54e-ef64b89e678b","Type":"ContainerStarted","Data":"f5f1e2dacbe0bf4412a66ef0adb8a7d51a8709297aee81fc9de163f13c9839d6"} Jul 25 02:00:18.495102 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:18.494839 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8" Jul 25 02:00:18.509175 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:18.509129 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8" podStartSLOduration=1.509116221 podStartE2EDuration="1.509116221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:00:18.508612425 +0000 UTC m=+2959.630098276" watchObservedRunningTime="2026-07-25 02:00:18.509116221 +0000 UTC m=+2959.630602071" Jul 25 02:00:23.499677 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:23.499604 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" Jul 25 02:00:29.501619 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:29.501587 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8" Jul 25 02:00:38.015007 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.014962 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk"] Jul 25 02:00:38.015454 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.015220 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" podUID="06812c3e-c7f7-4792-a0a4-0b44d7583334" containerName="maas-api" containerID="cri-o://97da332da80a6d06d7fc4a50661a2ac8cf5d3c04e293541f21d7f47e271ce0c6" gracePeriod=30 Jul 25 02:00:38.082665 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.082628 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8"] Jul 25 02:00:38.083634 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.083598 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8" podUID="d62cb034-3f64-4319-b54e-ef64b89e678b" containerName="payload-pre-processing" containerID="cri-o://771ed58ffd534bb563176c35465d19f51bb2cee13c760ce68c2380621e7b0e8c" gracePeriod=30 Jul 25 02:00:38.117899 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:00:38.117768 2554 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62cb034_3f64_4319_b54e_ef64b89e678b.slice/crio-771ed58ffd534bb563176c35465d19f51bb2cee13c760ce68c2380621e7b0e8c.scope\": RecentStats: unable to find data in memory cache]" Jul 25 02:00:38.338312 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.338286 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" Jul 25 02:00:38.367128 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.367099 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8" Jul 25 02:00:38.468298 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.468267 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06812c3e-c7f7-4792-a0a4-0b44d7583334-maas-api-tls\") pod \"06812c3e-c7f7-4792-a0a4-0b44d7583334\" (UID: \"06812c3e-c7f7-4792-a0a4-0b44d7583334\") " Jul 25 02:00:38.468467 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.468314 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzzz7\" (UniqueName: \"kubernetes.io/projected/d62cb034-3f64-4319-b54e-ef64b89e678b-kube-api-access-kzzz7\") pod \"d62cb034-3f64-4319-b54e-ef64b89e678b\" (UID: \"d62cb034-3f64-4319-b54e-ef64b89e678b\") " Jul 25 02:00:38.468467 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.468339 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w77wg\" (UniqueName: \"kubernetes.io/projected/06812c3e-c7f7-4792-a0a4-0b44d7583334-kube-api-access-w77wg\") pod \"06812c3e-c7f7-4792-a0a4-0b44d7583334\" (UID: \"06812c3e-c7f7-4792-a0a4-0b44d7583334\") " Jul 25 02:00:38.468467 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.468391 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62cb034-3f64-4319-b54e-ef64b89e678b-plugins-config-volume\") pod \"d62cb034-3f64-4319-b54e-ef64b89e678b\" (UID: \"d62cb034-3f64-4319-b54e-ef64b89e678b\") " Jul 25 02:00:38.468467 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.468436 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06812c3e-c7f7-4792-a0a4-0b44d7583334-openshift-service-ca\") pod \"06812c3e-c7f7-4792-a0a4-0b44d7583334\" (UID: \"06812c3e-c7f7-4792-a0a4-0b44d7583334\") " Jul 25 02:00:38.468896 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.468843 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62cb034-3f64-4319-b54e-ef64b89e678b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d62cb034-3f64-4319-b54e-ef64b89e678b" (UID: "d62cb034-3f64-4319-b54e-ef64b89e678b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:00:38.468896 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.468838 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06812c3e-c7f7-4792-a0a4-0b44d7583334-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "06812c3e-c7f7-4792-a0a4-0b44d7583334" (UID: "06812c3e-c7f7-4792-a0a4-0b44d7583334"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:00:38.470795 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.470762 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06812c3e-c7f7-4792-a0a4-0b44d7583334-kube-api-access-w77wg" (OuterVolumeSpecName: "kube-api-access-w77wg") pod "06812c3e-c7f7-4792-a0a4-0b44d7583334" (UID: "06812c3e-c7f7-4792-a0a4-0b44d7583334"). InnerVolumeSpecName "kube-api-access-w77wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:00:38.471184 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.471163 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62cb034-3f64-4319-b54e-ef64b89e678b-kube-api-access-kzzz7" (OuterVolumeSpecName: "kube-api-access-kzzz7") pod "d62cb034-3f64-4319-b54e-ef64b89e678b" (UID: "d62cb034-3f64-4319-b54e-ef64b89e678b"). InnerVolumeSpecName "kube-api-access-kzzz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:00:38.471287 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.471266 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06812c3e-c7f7-4792-a0a4-0b44d7583334-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "06812c3e-c7f7-4792-a0a4-0b44d7583334" (UID: "06812c3e-c7f7-4792-a0a4-0b44d7583334"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 02:00:38.569111 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.569039 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06812c3e-c7f7-4792-a0a4-0b44d7583334-openshift-service-ca\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:00:38.569111 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.569067 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06812c3e-c7f7-4792-a0a4-0b44d7583334-maas-api-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:00:38.569111 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.569080 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzzz7\" (UniqueName: \"kubernetes.io/projected/d62cb034-3f64-4319-b54e-ef64b89e678b-kube-api-access-kzzz7\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:00:38.569111 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.569094 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w77wg\" (UniqueName: \"kubernetes.io/projected/06812c3e-c7f7-4792-a0a4-0b44d7583334-kube-api-access-w77wg\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:00:38.569111 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.569106 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62cb034-3f64-4319-b54e-ef64b89e678b-plugins-config-volume\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:00:38.606482 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.606451 2554 generic.go:356] "Generic (PLEG): container finished" podID="06812c3e-c7f7-4792-a0a4-0b44d7583334" containerID="97da332da80a6d06d7fc4a50661a2ac8cf5d3c04e293541f21d7f47e271ce0c6" exitCode=0 Jul 25 02:00:38.606618 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.606497 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" Jul 25 02:00:38.606618 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.606536 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" event={"ID":"06812c3e-c7f7-4792-a0a4-0b44d7583334","Type":"ContainerDied","Data":"97da332da80a6d06d7fc4a50661a2ac8cf5d3c04e293541f21d7f47e271ce0c6"} Jul 25 02:00:38.606618 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.606586 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk" event={"ID":"06812c3e-c7f7-4792-a0a4-0b44d7583334","Type":"ContainerDied","Data":"6b2e31f3c86ebe478b7df44fd0adfdda60807cd332560548d3e1648194d95f2c"} Jul 25 02:00:38.606618 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.606607 2554 scope.go:117] "RemoveContainer" containerID="97da332da80a6d06d7fc4a50661a2ac8cf5d3c04e293541f21d7f47e271ce0c6" Jul 25 02:00:38.607993 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.607970 2554 generic.go:356] "Generic (PLEG): container finished" podID="d62cb034-3f64-4319-b54e-ef64b89e678b" containerID="771ed58ffd534bb563176c35465d19f51bb2cee13c760ce68c2380621e7b0e8c" exitCode=0 Jul 25 02:00:38.608071 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.608012 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8" event={"ID":"d62cb034-3f64-4319-b54e-ef64b89e678b","Type":"ContainerDied","Data":"771ed58ffd534bb563176c35465d19f51bb2cee13c760ce68c2380621e7b0e8c"} Jul 25 02:00:38.608071 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.608028 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8" Jul 25 02:00:38.608071 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.608061 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8" event={"ID":"d62cb034-3f64-4319-b54e-ef64b89e678b","Type":"ContainerDied","Data":"f5f1e2dacbe0bf4412a66ef0adb8a7d51a8709297aee81fc9de163f13c9839d6"} Jul 25 02:00:38.617549 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.617413 2554 scope.go:117] "RemoveContainer" containerID="97da332da80a6d06d7fc4a50661a2ac8cf5d3c04e293541f21d7f47e271ce0c6" Jul 25 02:00:38.617731 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:00:38.617705 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97da332da80a6d06d7fc4a50661a2ac8cf5d3c04e293541f21d7f47e271ce0c6\": container with ID starting with 97da332da80a6d06d7fc4a50661a2ac8cf5d3c04e293541f21d7f47e271ce0c6 not found: ID does not exist" containerID="97da332da80a6d06d7fc4a50661a2ac8cf5d3c04e293541f21d7f47e271ce0c6" Jul 25 02:00:38.617881 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.617742 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97da332da80a6d06d7fc4a50661a2ac8cf5d3c04e293541f21d7f47e271ce0c6"} err="failed to get container status \"97da332da80a6d06d7fc4a50661a2ac8cf5d3c04e293541f21d7f47e271ce0c6\": rpc error: code = NotFound desc = could not find container \"97da332da80a6d06d7fc4a50661a2ac8cf5d3c04e293541f21d7f47e271ce0c6\": container with ID starting with 97da332da80a6d06d7fc4a50661a2ac8cf5d3c04e293541f21d7f47e271ce0c6 not found: ID does not exist" Jul 25 02:00:38.617881 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.617764 2554 scope.go:117] "RemoveContainer" containerID="771ed58ffd534bb563176c35465d19f51bb2cee13c760ce68c2380621e7b0e8c" Jul 25 02:00:38.625959 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.625941 2554 scope.go:117] "RemoveContainer" containerID="771ed58ffd534bb563176c35465d19f51bb2cee13c760ce68c2380621e7b0e8c" Jul 25 02:00:38.626209 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:00:38.626192 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"771ed58ffd534bb563176c35465d19f51bb2cee13c760ce68c2380621e7b0e8c\": container with ID starting with 771ed58ffd534bb563176c35465d19f51bb2cee13c760ce68c2380621e7b0e8c not found: ID does not exist" containerID="771ed58ffd534bb563176c35465d19f51bb2cee13c760ce68c2380621e7b0e8c" Jul 25 02:00:38.626271 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.626216 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"771ed58ffd534bb563176c35465d19f51bb2cee13c760ce68c2380621e7b0e8c"} err="failed to get container status \"771ed58ffd534bb563176c35465d19f51bb2cee13c760ce68c2380621e7b0e8c\": rpc error: code = NotFound desc = could not find container \"771ed58ffd534bb563176c35465d19f51bb2cee13c760ce68c2380621e7b0e8c\": container with ID starting with 771ed58ffd534bb563176c35465d19f51bb2cee13c760ce68c2380621e7b0e8c not found: ID does not exist" Jul 25 02:00:38.632094 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.632073 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8"] Jul 25 02:00:38.637120 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.637100 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-17288039-5df6b9cb68-pzmm8"] Jul 25 02:00:38.646082 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.646056 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk"] Jul 25 02:00:38.649607 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:38.649589 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-17288039-76b5f87dc7-lrbdk"] Jul 25 02:00:39.382775 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:39.382694 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06812c3e-c7f7-4792-a0a4-0b44d7583334" path="/var/lib/kubelet/pods/06812c3e-c7f7-4792-a0a4-0b44d7583334/volumes" Jul 25 02:00:39.383156 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:00:39.383088 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d62cb034-3f64-4319-b54e-ef64b89e678b" path="/var/lib/kubelet/pods/d62cb034-3f64-4319-b54e-ef64b89e678b/volumes" Jul 25 02:01:11.747498 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:11.747463 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc"] Jul 25 02:01:11.748023 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:11.748007 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06812c3e-c7f7-4792-a0a4-0b44d7583334" containerName="maas-api" Jul 25 02:01:11.748069 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:11.748026 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="06812c3e-c7f7-4792-a0a4-0b44d7583334" containerName="maas-api" Jul 25 02:01:11.748069 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:11.748047 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d62cb034-3f64-4319-b54e-ef64b89e678b" containerName="payload-pre-processing" Jul 25 02:01:11.748069 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:11.748055 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d62cb034-3f64-4319-b54e-ef64b89e678b" containerName="payload-pre-processing" Jul 25 02:01:11.748177 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:11.748165 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="06812c3e-c7f7-4792-a0a4-0b44d7583334" containerName="maas-api" Jul 25 02:01:11.748212 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:11.748183 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d62cb034-3f64-4319-b54e-ef64b89e678b" containerName="payload-pre-processing" Jul 25 02:01:11.751674 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:11.751656 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" Jul 25 02:01:11.754653 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:11.754623 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-dc5514f6-serving-cert\"" Jul 25 02:01:11.761956 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:11.761933 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc"] Jul 25 02:01:11.763916 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:11.763890 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sf8k\" (UniqueName: \"kubernetes.io/projected/fec90f90-155f-4912-ae59-e20387a278cf-kube-api-access-4sf8k\") pod \"maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc\" (UID: \"fec90f90-155f-4912-ae59-e20387a278cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" Jul 25 02:01:11.764030 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:11.763947 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec90f90-155f-4912-ae59-e20387a278cf-openshift-service-ca\") pod \"maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc\" (UID: \"fec90f90-155f-4912-ae59-e20387a278cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" Jul 25 02:01:11.764030 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:11.764017 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec90f90-155f-4912-ae59-e20387a278cf-maas-api-tls\") pod \"maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc\" (UID: \"fec90f90-155f-4912-ae59-e20387a278cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" Jul 25 02:01:11.865094 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:11.865066 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sf8k\" (UniqueName: \"kubernetes.io/projected/fec90f90-155f-4912-ae59-e20387a278cf-kube-api-access-4sf8k\") pod \"maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc\" (UID: \"fec90f90-155f-4912-ae59-e20387a278cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" Jul 25 02:01:11.865256 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:11.865111 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec90f90-155f-4912-ae59-e20387a278cf-openshift-service-ca\") pod \"maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc\" (UID: \"fec90f90-155f-4912-ae59-e20387a278cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" Jul 25 02:01:11.865256 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:11.865148 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec90f90-155f-4912-ae59-e20387a278cf-maas-api-tls\") pod \"maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc\" (UID: \"fec90f90-155f-4912-ae59-e20387a278cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" Jul 25 02:01:11.865376 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:01:11.865260 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-serving-cert: secret "maas-api-e2e-mt-dc5514f6-serving-cert" not found Jul 25 02:01:11.865376 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:01:11.865336 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec90f90-155f-4912-ae59-e20387a278cf-maas-api-tls podName:fec90f90-155f-4912-ae59-e20387a278cf nodeName:}" failed. No retries permitted until 2026-07-25 02:01:12.365315464 +0000 UTC m=+3013.486801291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fec90f90-155f-4912-ae59-e20387a278cf-maas-api-tls") pod "maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" (UID: "fec90f90-155f-4912-ae59-e20387a278cf") : secret "maas-api-e2e-mt-dc5514f6-serving-cert" not found Jul 25 02:01:11.865774 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:11.865755 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec90f90-155f-4912-ae59-e20387a278cf-openshift-service-ca\") pod \"maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc\" (UID: \"fec90f90-155f-4912-ae59-e20387a278cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" Jul 25 02:01:11.873559 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:11.873532 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sf8k\" (UniqueName: \"kubernetes.io/projected/fec90f90-155f-4912-ae59-e20387a278cf-kube-api-access-4sf8k\") pod \"maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc\" (UID: \"fec90f90-155f-4912-ae59-e20387a278cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" Jul 25 02:01:12.369487 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:12.369438 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec90f90-155f-4912-ae59-e20387a278cf-maas-api-tls\") pod \"maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc\" (UID: \"fec90f90-155f-4912-ae59-e20387a278cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" Jul 25 02:01:12.372571 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:12.372537 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec90f90-155f-4912-ae59-e20387a278cf-maas-api-tls\") pod \"maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc\" (UID: \"fec90f90-155f-4912-ae59-e20387a278cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" Jul 25 02:01:12.663202 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:12.663173 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" Jul 25 02:01:12.805100 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:12.805074 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc"] Jul 25 02:01:12.807166 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:01:12.807137 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec90f90_155f_4912_ae59_e20387a278cf.slice/crio-ae6bb6debb0d3ba6fa88a184f0452213208d13bc2f593297b7de4c7f7bc8bd15 WatchSource:0}: Error finding container ae6bb6debb0d3ba6fa88a184f0452213208d13bc2f593297b7de4c7f7bc8bd15: Status 404 returned error can't find the container with id ae6bb6debb0d3ba6fa88a184f0452213208d13bc2f593297b7de4c7f7bc8bd15 Jul 25 02:01:13.799180 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:13.799140 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" event={"ID":"fec90f90-155f-4912-ae59-e20387a278cf","Type":"ContainerStarted","Data":"4b0eccf2139e6c17ac27d5c8c331786e5a12914176208551d5d46ecb9ddc5f12"} Jul 25 02:01:13.799387 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:13.799202 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" event={"ID":"fec90f90-155f-4912-ae59-e20387a278cf","Type":"ContainerStarted","Data":"ae6bb6debb0d3ba6fa88a184f0452213208d13bc2f593297b7de4c7f7bc8bd15"} Jul 25 02:01:13.799387 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:13.799239 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" Jul 25 02:01:13.814670 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:13.814626 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" podStartSLOduration=2.352910655 podStartE2EDuration="2.8146154s" totalImagesPullingTime="461.704745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:01:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:01:13.813537783 +0000 UTC m=+3014.935023634" watchObservedRunningTime="2026-07-25 02:01:13.8146154 +0000 UTC m=+3014.936101250" Jul 25 02:01:19.809658 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:01:19.809631 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" Jul 25 02:02:00.953361 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:00.953284 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc"] Jul 25 02:02:00.953880 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:00.953507 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" podUID="fec90f90-155f-4912-ae59-e20387a278cf" containerName="maas-api" containerID="cri-o://4b0eccf2139e6c17ac27d5c8c331786e5a12914176208551d5d46ecb9ddc5f12" gracePeriod=30 Jul 25 02:02:01.203311 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:01.203289 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" Jul 25 02:02:01.382628 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:01.382550 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sf8k\" (UniqueName: \"kubernetes.io/projected/fec90f90-155f-4912-ae59-e20387a278cf-kube-api-access-4sf8k\") pod \"fec90f90-155f-4912-ae59-e20387a278cf\" (UID: \"fec90f90-155f-4912-ae59-e20387a278cf\") " Jul 25 02:02:01.382628 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:01.382619 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec90f90-155f-4912-ae59-e20387a278cf-maas-api-tls\") pod \"fec90f90-155f-4912-ae59-e20387a278cf\" (UID: \"fec90f90-155f-4912-ae59-e20387a278cf\") " Jul 25 02:02:01.383027 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:01.382671 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec90f90-155f-4912-ae59-e20387a278cf-openshift-service-ca\") pod \"fec90f90-155f-4912-ae59-e20387a278cf\" (UID: \"fec90f90-155f-4912-ae59-e20387a278cf\") " Jul 25 02:02:01.383150 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:01.383117 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec90f90-155f-4912-ae59-e20387a278cf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fec90f90-155f-4912-ae59-e20387a278cf" (UID: "fec90f90-155f-4912-ae59-e20387a278cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:02:01.385001 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:01.384980 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec90f90-155f-4912-ae59-e20387a278cf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fec90f90-155f-4912-ae59-e20387a278cf" (UID: "fec90f90-155f-4912-ae59-e20387a278cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 02:02:01.385230 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:01.385213 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec90f90-155f-4912-ae59-e20387a278cf-kube-api-access-4sf8k" (OuterVolumeSpecName: "kube-api-access-4sf8k") pod "fec90f90-155f-4912-ae59-e20387a278cf" (UID: "fec90f90-155f-4912-ae59-e20387a278cf"). InnerVolumeSpecName "kube-api-access-4sf8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:02:01.489652 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:01.484296 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sf8k\" (UniqueName: \"kubernetes.io/projected/fec90f90-155f-4912-ae59-e20387a278cf-kube-api-access-4sf8k\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:02:01.489652 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:01.484328 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec90f90-155f-4912-ae59-e20387a278cf-maas-api-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:02:01.489652 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:01.484343 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec90f90-155f-4912-ae59-e20387a278cf-openshift-service-ca\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:02:02.059301 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:02.059263 2554 generic.go:356] "Generic (PLEG): container finished" podID="fec90f90-155f-4912-ae59-e20387a278cf" containerID="4b0eccf2139e6c17ac27d5c8c331786e5a12914176208551d5d46ecb9ddc5f12" exitCode=0 Jul 25 02:02:02.059712 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:02.059319 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" Jul 25 02:02:02.059712 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:02.059327 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" event={"ID":"fec90f90-155f-4912-ae59-e20387a278cf","Type":"ContainerDied","Data":"4b0eccf2139e6c17ac27d5c8c331786e5a12914176208551d5d46ecb9ddc5f12"} Jul 25 02:02:02.059712 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:02.059392 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc" event={"ID":"fec90f90-155f-4912-ae59-e20387a278cf","Type":"ContainerDied","Data":"ae6bb6debb0d3ba6fa88a184f0452213208d13bc2f593297b7de4c7f7bc8bd15"} Jul 25 02:02:02.059712 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:02.059412 2554 scope.go:117] "RemoveContainer" containerID="4b0eccf2139e6c17ac27d5c8c331786e5a12914176208551d5d46ecb9ddc5f12" Jul 25 02:02:02.069128 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:02.069110 2554 scope.go:117] "RemoveContainer" containerID="4b0eccf2139e6c17ac27d5c8c331786e5a12914176208551d5d46ecb9ddc5f12" Jul 25 02:02:02.069464 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:02:02.069432 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b0eccf2139e6c17ac27d5c8c331786e5a12914176208551d5d46ecb9ddc5f12\": container with ID starting with 4b0eccf2139e6c17ac27d5c8c331786e5a12914176208551d5d46ecb9ddc5f12 not found: ID does not exist" containerID="4b0eccf2139e6c17ac27d5c8c331786e5a12914176208551d5d46ecb9ddc5f12" Jul 25 02:02:02.069572 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:02.069474 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b0eccf2139e6c17ac27d5c8c331786e5a12914176208551d5d46ecb9ddc5f12"} err="failed to get container status \"4b0eccf2139e6c17ac27d5c8c331786e5a12914176208551d5d46ecb9ddc5f12\": rpc error: code = NotFound desc = could not find container \"4b0eccf2139e6c17ac27d5c8c331786e5a12914176208551d5d46ecb9ddc5f12\": container with ID starting with 4b0eccf2139e6c17ac27d5c8c331786e5a12914176208551d5d46ecb9ddc5f12 not found: ID does not exist" Jul 25 02:02:02.082710 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:02.082683 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc"] Jul 25 02:02:02.084662 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:02.084644 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc5514f6-85874fcf7c-hckmc"] Jul 25 02:02:03.383157 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:02:03.383124 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fec90f90-155f-4912-ae59-e20387a278cf" path="/var/lib/kubelet/pods/fec90f90-155f-4912-ae59-e20387a278cf/volumes" Jul 25 02:03:01.087714 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:01.087672 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qqf84"] Jul 25 02:03:01.088252 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:01.088231 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fec90f90-155f-4912-ae59-e20387a278cf" containerName="maas-api" Jul 25 02:03:01.088336 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:01.088255 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fec90f90-155f-4912-ae59-e20387a278cf" containerName="maas-api" Jul 25 02:03:01.088389 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:01.088354 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fec90f90-155f-4912-ae59-e20387a278cf" containerName="maas-api" Jul 25 02:03:01.092717 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:01.092696 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qqf84" Jul 25 02:03:01.101404 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:01.101373 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qqf84"] Jul 25 02:03:01.174624 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:01.174594 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbljw\" (UniqueName: \"kubernetes.io/projected/74692fb2-f356-44ad-aaa8-db741f5aeb91-kube-api-access-qbljw\") pod \"kuadrant-operator-catalog-qqf84\" (UID: \"74692fb2-f356-44ad-aaa8-db741f5aeb91\") " pod="kuadrant-system/kuadrant-operator-catalog-qqf84" Jul 25 02:03:01.276119 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:01.276078 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbljw\" (UniqueName: \"kubernetes.io/projected/74692fb2-f356-44ad-aaa8-db741f5aeb91-kube-api-access-qbljw\") pod \"kuadrant-operator-catalog-qqf84\" (UID: \"74692fb2-f356-44ad-aaa8-db741f5aeb91\") " pod="kuadrant-system/kuadrant-operator-catalog-qqf84" Jul 25 02:03:01.286830 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:01.286779 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbljw\" (UniqueName: \"kubernetes.io/projected/74692fb2-f356-44ad-aaa8-db741f5aeb91-kube-api-access-qbljw\") pod \"kuadrant-operator-catalog-qqf84\" (UID: \"74692fb2-f356-44ad-aaa8-db741f5aeb91\") " pod="kuadrant-system/kuadrant-operator-catalog-qqf84" Jul 25 02:03:01.404284 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:01.404214 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qqf84" Jul 25 02:03:01.541472 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:01.541440 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qqf84"] Jul 25 02:03:01.544628 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:03:01.544604 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74692fb2_f356_44ad_aaa8_db741f5aeb91.slice/crio-21c0a4c028c14d5972bfde8204d98fb05d4f35266b207d45080d11fb98f6fb43 WatchSource:0}: Error finding container 21c0a4c028c14d5972bfde8204d98fb05d4f35266b207d45080d11fb98f6fb43: Status 404 returned error can't find the container with id 21c0a4c028c14d5972bfde8204d98fb05d4f35266b207d45080d11fb98f6fb43 Jul 25 02:03:02.372329 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:02.372246 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qqf84" event={"ID":"74692fb2-f356-44ad-aaa8-db741f5aeb91","Type":"ContainerStarted","Data":"443f94db26e000fb02d11665d6c68358dcc3db71ba10d947ef9dbdf4ec05490c"} Jul 25 02:03:02.372329 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:02.372305 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qqf84" event={"ID":"74692fb2-f356-44ad-aaa8-db741f5aeb91","Type":"ContainerStarted","Data":"21c0a4c028c14d5972bfde8204d98fb05d4f35266b207d45080d11fb98f6fb43"} Jul 25 02:03:02.385986 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:02.385938 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-qqf84" podStartSLOduration=0.8450245 podStartE2EDuration="1.385923298s" totalImagesPullingTime="540.898798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:03:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:03:02.385174002 +0000 UTC m=+3123.506659853" watchObservedRunningTime="2026-07-25 02:03:02.385923298 +0000 UTC m=+3123.507409147" Jul 25 02:03:11.404921 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:11.404887 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-qqf84" Jul 25 02:03:11.405299 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:11.404951 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-qqf84" Jul 25 02:03:11.427306 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:11.427285 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-qqf84" Jul 25 02:03:11.449022 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:11.449000 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-qqf84" Jul 25 02:03:11.654825 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:11.654775 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qqf84"] Jul 25 02:03:13.429623 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:13.429585 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-qqf84" podUID="74692fb2-f356-44ad-aaa8-db741f5aeb91" containerName="registry-server" containerID="cri-o://443f94db26e000fb02d11665d6c68358dcc3db71ba10d947ef9dbdf4ec05490c" gracePeriod=2 Jul 25 02:03:13.688033 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:13.687977 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qqf84" Jul 25 02:03:13.779983 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:13.779953 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbljw\" (UniqueName: \"kubernetes.io/projected/74692fb2-f356-44ad-aaa8-db741f5aeb91-kube-api-access-qbljw\") pod \"74692fb2-f356-44ad-aaa8-db741f5aeb91\" (UID: \"74692fb2-f356-44ad-aaa8-db741f5aeb91\") " Jul 25 02:03:13.782273 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:13.782233 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74692fb2-f356-44ad-aaa8-db741f5aeb91-kube-api-access-qbljw" (OuterVolumeSpecName: "kube-api-access-qbljw") pod "74692fb2-f356-44ad-aaa8-db741f5aeb91" (UID: "74692fb2-f356-44ad-aaa8-db741f5aeb91"). InnerVolumeSpecName "kube-api-access-qbljw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:03:13.881414 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:13.881381 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbljw\" (UniqueName: \"kubernetes.io/projected/74692fb2-f356-44ad-aaa8-db741f5aeb91-kube-api-access-qbljw\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:03:14.436032 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:14.435994 2554 generic.go:356] "Generic (PLEG): container finished" podID="74692fb2-f356-44ad-aaa8-db741f5aeb91" containerID="443f94db26e000fb02d11665d6c68358dcc3db71ba10d947ef9dbdf4ec05490c" exitCode=0 Jul 25 02:03:14.436512 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:14.436046 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qqf84" Jul 25 02:03:14.436512 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:14.436082 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qqf84" event={"ID":"74692fb2-f356-44ad-aaa8-db741f5aeb91","Type":"ContainerDied","Data":"443f94db26e000fb02d11665d6c68358dcc3db71ba10d947ef9dbdf4ec05490c"} Jul 25 02:03:14.436512 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:14.436132 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qqf84" event={"ID":"74692fb2-f356-44ad-aaa8-db741f5aeb91","Type":"ContainerDied","Data":"21c0a4c028c14d5972bfde8204d98fb05d4f35266b207d45080d11fb98f6fb43"} Jul 25 02:03:14.436512 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:14.436151 2554 scope.go:117] "RemoveContainer" containerID="443f94db26e000fb02d11665d6c68358dcc3db71ba10d947ef9dbdf4ec05490c" Jul 25 02:03:14.445688 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:14.445665 2554 scope.go:117] "RemoveContainer" containerID="443f94db26e000fb02d11665d6c68358dcc3db71ba10d947ef9dbdf4ec05490c" Jul 25 02:03:14.445996 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:03:14.445977 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443f94db26e000fb02d11665d6c68358dcc3db71ba10d947ef9dbdf4ec05490c\": container with ID starting with 443f94db26e000fb02d11665d6c68358dcc3db71ba10d947ef9dbdf4ec05490c not found: ID does not exist" containerID="443f94db26e000fb02d11665d6c68358dcc3db71ba10d947ef9dbdf4ec05490c" Jul 25 02:03:14.446083 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:14.446008 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443f94db26e000fb02d11665d6c68358dcc3db71ba10d947ef9dbdf4ec05490c"} err="failed to get container status \"443f94db26e000fb02d11665d6c68358dcc3db71ba10d947ef9dbdf4ec05490c\": rpc error: code = NotFound desc = could not find container \"443f94db26e000fb02d11665d6c68358dcc3db71ba10d947ef9dbdf4ec05490c\": container with ID starting with 443f94db26e000fb02d11665d6c68358dcc3db71ba10d947ef9dbdf4ec05490c not found: ID does not exist" Jul 25 02:03:14.458382 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:14.458361 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qqf84"] Jul 25 02:03:14.463480 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:14.463461 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qqf84"] Jul 25 02:03:15.381340 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:15.381309 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74692fb2-f356-44ad-aaa8-db741f5aeb91" path="/var/lib/kubelet/pods/74692fb2-f356-44ad-aaa8-db741f5aeb91/volumes" Jul 25 02:03:55.731553 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:55.731509 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd"] Jul 25 02:03:55.732156 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:55.732133 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74692fb2-f356-44ad-aaa8-db741f5aeb91" containerName="registry-server" Jul 25 02:03:55.732239 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:55.732180 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="74692fb2-f356-44ad-aaa8-db741f5aeb91" containerName="registry-server" Jul 25 02:03:55.732296 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:55.732281 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="74692fb2-f356-44ad-aaa8-db741f5aeb91" containerName="registry-server" Jul 25 02:03:55.735610 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:55.735584 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" Jul 25 02:03:55.737757 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:55.737734 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c13f56-serving-cert\"" Jul 25 02:03:55.745016 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:55.744447 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd"] Jul 25 02:03:55.837848 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:55.837818 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4766c768-f7c0-4af1-af14-56c55831c4f1-maas-api-tls\") pod \"maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd\" (UID: \"4766c768-f7c0-4af1-af14-56c55831c4f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" Jul 25 02:03:55.838035 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:55.837860 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4766c768-f7c0-4af1-af14-56c55831c4f1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd\" (UID: \"4766c768-f7c0-4af1-af14-56c55831c4f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" Jul 25 02:03:55.838035 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:55.837886 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g4nn\" (UniqueName: \"kubernetes.io/projected/4766c768-f7c0-4af1-af14-56c55831c4f1-kube-api-access-8g4nn\") pod \"maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd\" (UID: \"4766c768-f7c0-4af1-af14-56c55831c4f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" Jul 25 02:03:55.938975 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:55.938934 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4766c768-f7c0-4af1-af14-56c55831c4f1-maas-api-tls\") pod \"maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd\" (UID: \"4766c768-f7c0-4af1-af14-56c55831c4f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" Jul 25 02:03:55.939149 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:55.938991 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4766c768-f7c0-4af1-af14-56c55831c4f1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd\" (UID: \"4766c768-f7c0-4af1-af14-56c55831c4f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" Jul 25 02:03:55.939149 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:55.939030 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g4nn\" (UniqueName: \"kubernetes.io/projected/4766c768-f7c0-4af1-af14-56c55831c4f1-kube-api-access-8g4nn\") pod \"maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd\" (UID: \"4766c768-f7c0-4af1-af14-56c55831c4f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" Jul 25 02:03:55.939149 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:03:55.939091 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-serving-cert: secret "maas-api-e2e-inf-a-c13f56-serving-cert" not found Jul 25 02:03:55.939262 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:03:55.939176 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4766c768-f7c0-4af1-af14-56c55831c4f1-maas-api-tls podName:4766c768-f7c0-4af1-af14-56c55831c4f1 nodeName:}" failed. No retries permitted until 2026-07-25 02:03:56.439153677 +0000 UTC m=+3177.560639506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4766c768-f7c0-4af1-af14-56c55831c4f1-maas-api-tls") pod "maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" (UID: "4766c768-f7c0-4af1-af14-56c55831c4f1") : secret "maas-api-e2e-inf-a-c13f56-serving-cert" not found Jul 25 02:03:55.939729 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:55.939713 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4766c768-f7c0-4af1-af14-56c55831c4f1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd\" (UID: \"4766c768-f7c0-4af1-af14-56c55831c4f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" Jul 25 02:03:55.948074 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:55.948043 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g4nn\" (UniqueName: \"kubernetes.io/projected/4766c768-f7c0-4af1-af14-56c55831c4f1-kube-api-access-8g4nn\") pod \"maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd\" (UID: \"4766c768-f7c0-4af1-af14-56c55831c4f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" Jul 25 02:03:56.445029 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:56.444999 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4766c768-f7c0-4af1-af14-56c55831c4f1-maas-api-tls\") pod \"maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd\" (UID: \"4766c768-f7c0-4af1-af14-56c55831c4f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" Jul 25 02:03:56.447454 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:56.447429 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4766c768-f7c0-4af1-af14-56c55831c4f1-maas-api-tls\") pod \"maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd\" (UID: \"4766c768-f7c0-4af1-af14-56c55831c4f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" Jul 25 02:03:56.649616 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:56.649584 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" Jul 25 02:03:56.789416 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:56.789391 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd"] Jul 25 02:03:56.793826 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:03:56.791671 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4766c768_f7c0_4af1_af14_56c55831c4f1.slice/crio-f3c6324370e440cb5cada8197a426682a594f4f13e226dfb8cd0f9d8967ae868 WatchSource:0}: Error finding container f3c6324370e440cb5cada8197a426682a594f4f13e226dfb8cd0f9d8967ae868: Status 404 returned error can't find the container with id f3c6324370e440cb5cada8197a426682a594f4f13e226dfb8cd0f9d8967ae868 Jul 25 02:03:57.668996 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:57.668961 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" event={"ID":"4766c768-f7c0-4af1-af14-56c55831c4f1","Type":"ContainerStarted","Data":"1fdb8cab42a5776859d70d8bb130fe9d5482570e0ac6aa93f3ea1faae5301dfa"} Jul 25 02:03:57.669198 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:57.669023 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" event={"ID":"4766c768-f7c0-4af1-af14-56c55831c4f1","Type":"ContainerStarted","Data":"f3c6324370e440cb5cada8197a426682a594f4f13e226dfb8cd0f9d8967ae868"} Jul 25 02:03:57.669198 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:57.669112 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" Jul 25 02:03:57.685046 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:03:57.684968 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" podStartSLOduration=2.225336233 podStartE2EDuration="2.684953198s" totalImagesPullingTime="459.616965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:03:57.684341738 +0000 UTC m=+3178.805827608" watchObservedRunningTime="2026-07-25 02:03:57.684953198 +0000 UTC m=+3178.806439049" Jul 25 02:04:03.589519 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:03.589484 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px"] Jul 25 02:04:03.594501 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:03.594482 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" Jul 25 02:04:03.596916 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:03.596894 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1a3abd-serving-cert\"" Jul 25 02:04:03.612144 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:03.612120 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px"] Jul 25 02:04:03.613986 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:03.613960 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjdq6\" (UniqueName: \"kubernetes.io/projected/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-kube-api-access-sjdq6\") pod \"maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px\" (UID: \"0163bfc9-2ca0-43bf-9d06-1f56db5bc728\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" Jul 25 02:04:03.614089 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:03.614004 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-maas-api-tls\") pod \"maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px\" (UID: \"0163bfc9-2ca0-43bf-9d06-1f56db5bc728\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" Jul 25 02:04:03.614141 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:03.614125 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px\" (UID: \"0163bfc9-2ca0-43bf-9d06-1f56db5bc728\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" Jul 25 02:04:03.679721 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:03.679689 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" Jul 25 02:04:03.714978 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:03.714941 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px\" (UID: \"0163bfc9-2ca0-43bf-9d06-1f56db5bc728\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" Jul 25 02:04:03.715122 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:03.715023 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjdq6\" (UniqueName: \"kubernetes.io/projected/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-kube-api-access-sjdq6\") pod \"maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px\" (UID: \"0163bfc9-2ca0-43bf-9d06-1f56db5bc728\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" Jul 25 02:04:03.715122 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:03.715067 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-maas-api-tls\") pod \"maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px\" (UID: \"0163bfc9-2ca0-43bf-9d06-1f56db5bc728\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" Jul 25 02:04:03.715362 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:04:03.715340 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-serving-cert: secret "maas-api-e2e-inf-b-1a3abd-serving-cert" not found Jul 25 02:04:03.715447 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:04:03.715421 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-maas-api-tls podName:0163bfc9-2ca0-43bf-9d06-1f56db5bc728 nodeName:}" failed. No retries permitted until 2026-07-25 02:04:04.215398977 +0000 UTC m=+3185.336884807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-maas-api-tls") pod "maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" (UID: "0163bfc9-2ca0-43bf-9d06-1f56db5bc728") : secret "maas-api-e2e-inf-b-1a3abd-serving-cert" not found Jul 25 02:04:03.715739 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:03.715720 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px\" (UID: \"0163bfc9-2ca0-43bf-9d06-1f56db5bc728\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" Jul 25 02:04:03.743649 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:03.743622 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjdq6\" (UniqueName: \"kubernetes.io/projected/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-kube-api-access-sjdq6\") pod \"maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px\" (UID: \"0163bfc9-2ca0-43bf-9d06-1f56db5bc728\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" Jul 25 02:04:04.219165 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.219130 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-maas-api-tls\") pod \"maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px\" (UID: \"0163bfc9-2ca0-43bf-9d06-1f56db5bc728\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" Jul 25 02:04:04.221616 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.221596 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-maas-api-tls\") pod \"maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px\" (UID: \"0163bfc9-2ca0-43bf-9d06-1f56db5bc728\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" Jul 25 02:04:04.412266 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.412227 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm"] Jul 25 02:04:04.416004 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.415988 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm" Jul 25 02:04:04.417909 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.417886 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1a3abd\"" Jul 25 02:04:04.417909 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.417901 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1a3abd-dockercfg-dpbw2\"" Jul 25 02:04:04.424793 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.424770 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm"] Jul 25 02:04:04.433704 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.433674 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv"] Jul 25 02:04:04.437170 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.437151 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv" Jul 25 02:04:04.449201 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.449176 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv"] Jul 25 02:04:04.506290 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.506217 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" Jul 25 02:04:04.522162 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.521721 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf2hq\" (UniqueName: \"kubernetes.io/projected/75573941-78c1-4af4-b33f-83f31cbcde66-kube-api-access-bf2hq\") pod \"payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv\" (UID: \"75573941-78c1-4af4-b33f-83f31cbcde66\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv" Jul 25 02:04:04.522162 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.521832 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76baa294-0c16-408c-b9a8-cd44e621ca1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm\" (UID: \"76baa294-0c16-408c-b9a8-cd44e621ca1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm" Jul 25 02:04:04.522162 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.521878 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd7cs\" (UniqueName: \"kubernetes.io/projected/76baa294-0c16-408c-b9a8-cd44e621ca1f-kube-api-access-bd7cs\") pod \"payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm\" (UID: \"76baa294-0c16-408c-b9a8-cd44e621ca1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm" Jul 25 02:04:04.522162 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.521980 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75573941-78c1-4af4-b33f-83f31cbcde66-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv\" (UID: \"75573941-78c1-4af4-b33f-83f31cbcde66\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv" Jul 25 02:04:04.623351 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.623320 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76baa294-0c16-408c-b9a8-cd44e621ca1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm\" (UID: \"76baa294-0c16-408c-b9a8-cd44e621ca1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm" Jul 25 02:04:04.623828 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.623363 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd7cs\" (UniqueName: \"kubernetes.io/projected/76baa294-0c16-408c-b9a8-cd44e621ca1f-kube-api-access-bd7cs\") pod \"payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm\" (UID: \"76baa294-0c16-408c-b9a8-cd44e621ca1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm" Jul 25 02:04:04.623828 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.623453 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75573941-78c1-4af4-b33f-83f31cbcde66-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv\" (UID: \"75573941-78c1-4af4-b33f-83f31cbcde66\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv" Jul 25 02:04:04.623828 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.623545 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf2hq\" (UniqueName: \"kubernetes.io/projected/75573941-78c1-4af4-b33f-83f31cbcde66-kube-api-access-bf2hq\") pod \"payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv\" (UID: \"75573941-78c1-4af4-b33f-83f31cbcde66\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv" Jul 25 02:04:04.624385 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.624362 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76baa294-0c16-408c-b9a8-cd44e621ca1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm\" (UID: \"76baa294-0c16-408c-b9a8-cd44e621ca1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm" Jul 25 02:04:04.624484 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.624440 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75573941-78c1-4af4-b33f-83f31cbcde66-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv\" (UID: \"75573941-78c1-4af4-b33f-83f31cbcde66\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv" Jul 25 02:04:04.631117 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.631095 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf2hq\" (UniqueName: \"kubernetes.io/projected/75573941-78c1-4af4-b33f-83f31cbcde66-kube-api-access-bf2hq\") pod \"payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv\" (UID: \"75573941-78c1-4af4-b33f-83f31cbcde66\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv" Jul 25 02:04:04.631245 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.631124 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd7cs\" (UniqueName: \"kubernetes.io/projected/76baa294-0c16-408c-b9a8-cd44e621ca1f-kube-api-access-bd7cs\") pod \"payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm\" (UID: \"76baa294-0c16-408c-b9a8-cd44e621ca1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm" Jul 25 02:04:04.644627 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.644609 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px"] Jul 25 02:04:04.646466 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:04:04.646444 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0163bfc9_2ca0_43bf_9d06_1f56db5bc728.slice/crio-9220d162ad5a49a99b0af21edce15d4bd0da186817675f2740141bd8f3fda6cb WatchSource:0}: Error finding container 9220d162ad5a49a99b0af21edce15d4bd0da186817675f2740141bd8f3fda6cb: Status 404 returned error can't find the container with id 9220d162ad5a49a99b0af21edce15d4bd0da186817675f2740141bd8f3fda6cb Jul 25 02:04:04.709120 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.709092 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" event={"ID":"0163bfc9-2ca0-43bf-9d06-1f56db5bc728","Type":"ContainerStarted","Data":"9220d162ad5a49a99b0af21edce15d4bd0da186817675f2740141bd8f3fda6cb"} Jul 25 02:04:04.728749 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.728715 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm" Jul 25 02:04:04.747620 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.747574 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv" Jul 25 02:04:04.906961 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:04.906936 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv"] Jul 25 02:04:04.908964 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:04:04.908937 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75573941_78c1_4af4_b33f_83f31cbcde66.slice/crio-a9fc89a9ddd8324ef96720988f4a0e0fef19dcd0a56ba30668037496955815c7 WatchSource:0}: Error finding container a9fc89a9ddd8324ef96720988f4a0e0fef19dcd0a56ba30668037496955815c7: Status 404 returned error can't find the container with id a9fc89a9ddd8324ef96720988f4a0e0fef19dcd0a56ba30668037496955815c7 Jul 25 02:04:05.084456 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:05.084434 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm"] Jul 25 02:04:05.086780 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:04:05.086756 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76baa294_0c16_408c_b9a8_cd44e621ca1f.slice/crio-1aedae3b5f974d5e32c6d2b27a8c4e02dc19991422b9b74f5af7d333375bfb16 WatchSource:0}: Error finding container 1aedae3b5f974d5e32c6d2b27a8c4e02dc19991422b9b74f5af7d333375bfb16: Status 404 returned error can't find the container with id 1aedae3b5f974d5e32c6d2b27a8c4e02dc19991422b9b74f5af7d333375bfb16 Jul 25 02:04:05.716706 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:05.716666 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm" event={"ID":"76baa294-0c16-408c-b9a8-cd44e621ca1f","Type":"ContainerStarted","Data":"ac708ba81d8a15bd2c331669f956e9271431455872247dbfd938a2a715a50fdd"} Jul 25 02:04:05.717185 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:05.716734 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm" event={"ID":"76baa294-0c16-408c-b9a8-cd44e621ca1f","Type":"ContainerStarted","Data":"1aedae3b5f974d5e32c6d2b27a8c4e02dc19991422b9b74f5af7d333375bfb16"} Jul 25 02:04:05.717185 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:05.716762 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm" Jul 25 02:04:05.718288 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:05.718264 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" event={"ID":"0163bfc9-2ca0-43bf-9d06-1f56db5bc728","Type":"ContainerStarted","Data":"be77c77846949f6c09ce2b7eded511baf11de1d35a985959f9c72c1a6c8e7fe7"} Jul 25 02:04:05.718411 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:05.718340 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" Jul 25 02:04:05.719943 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:05.719923 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv" event={"ID":"75573941-78c1-4af4-b33f-83f31cbcde66","Type":"ContainerStarted","Data":"6c3ec906d2555049604135f9da9d6dc05922b676c544d128026f93b09240af0b"} Jul 25 02:04:05.720045 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:05.719964 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv" event={"ID":"75573941-78c1-4af4-b33f-83f31cbcde66","Type":"ContainerStarted","Data":"a9fc89a9ddd8324ef96720988f4a0e0fef19dcd0a56ba30668037496955815c7"} Jul 25 02:04:05.720045 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:05.720006 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv" Jul 25 02:04:05.730852 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:05.730786 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm" podStartSLOduration=1.730773332 podStartE2EDuration="1.730773332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:04:05.729495466 +0000 UTC m=+3186.850981316" watchObservedRunningTime="2026-07-25 02:04:05.730773332 +0000 UTC m=+3186.852259161" Jul 25 02:04:05.743042 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:05.743005 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv" podStartSLOduration=1.742988435 podStartE2EDuration="1.742988435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:04:05.741662699 +0000 UTC m=+3186.863148549" watchObservedRunningTime="2026-07-25 02:04:05.742988435 +0000 UTC m=+3186.864474285" Jul 25 02:04:05.755297 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:05.755262 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" podStartSLOduration=2.362730724 podStartE2EDuration="2.755251484s" totalImagesPullingTime="392.52076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:04:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:04:05.754478614 +0000 UTC m=+3186.875964463" watchObservedRunningTime="2026-07-25 02:04:05.755251484 +0000 UTC m=+3186.876737333" Jul 25 02:04:11.730832 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:11.730784 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" Jul 25 02:04:16.727636 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:16.727605 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv" Jul 25 02:04:16.728047 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:04:16.727781 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm" Jul 25 02:05:44.736411 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:44.736378 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd"] Jul 25 02:05:44.736999 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:44.736631 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" podUID="4766c768-f7c0-4af1-af14-56c55831c4f1" containerName="maas-api" containerID="cri-o://1fdb8cab42a5776859d70d8bb130fe9d5482570e0ac6aa93f3ea1faae5301dfa" gracePeriod=30 Jul 25 02:05:44.981488 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:44.981465 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" Jul 25 02:05:45.069876 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:45.069779 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8g4nn\" (UniqueName: \"kubernetes.io/projected/4766c768-f7c0-4af1-af14-56c55831c4f1-kube-api-access-8g4nn\") pod \"4766c768-f7c0-4af1-af14-56c55831c4f1\" (UID: \"4766c768-f7c0-4af1-af14-56c55831c4f1\") " Jul 25 02:05:45.069876 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:45.069871 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4766c768-f7c0-4af1-af14-56c55831c4f1-openshift-service-ca\") pod \"4766c768-f7c0-4af1-af14-56c55831c4f1\" (UID: \"4766c768-f7c0-4af1-af14-56c55831c4f1\") " Jul 25 02:05:45.070083 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:45.069937 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4766c768-f7c0-4af1-af14-56c55831c4f1-maas-api-tls\") pod \"4766c768-f7c0-4af1-af14-56c55831c4f1\" (UID: \"4766c768-f7c0-4af1-af14-56c55831c4f1\") " Jul 25 02:05:45.070267 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:45.070236 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4766c768-f7c0-4af1-af14-56c55831c4f1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4766c768-f7c0-4af1-af14-56c55831c4f1" (UID: "4766c768-f7c0-4af1-af14-56c55831c4f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:05:45.072224 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:45.072192 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4766c768-f7c0-4af1-af14-56c55831c4f1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4766c768-f7c0-4af1-af14-56c55831c4f1" (UID: "4766c768-f7c0-4af1-af14-56c55831c4f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 02:05:45.072527 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:45.072507 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4766c768-f7c0-4af1-af14-56c55831c4f1-kube-api-access-8g4nn" (OuterVolumeSpecName: "kube-api-access-8g4nn") pod "4766c768-f7c0-4af1-af14-56c55831c4f1" (UID: "4766c768-f7c0-4af1-af14-56c55831c4f1"). InnerVolumeSpecName "kube-api-access-8g4nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:05:45.171380 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:45.171344 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4766c768-f7c0-4af1-af14-56c55831c4f1-maas-api-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:05:45.171380 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:45.171379 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g4nn\" (UniqueName: \"kubernetes.io/projected/4766c768-f7c0-4af1-af14-56c55831c4f1-kube-api-access-8g4nn\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:05:45.171551 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:45.171396 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4766c768-f7c0-4af1-af14-56c55831c4f1-openshift-service-ca\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:05:45.256291 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:45.256260 2554 generic.go:356] "Generic (PLEG): container finished" podID="4766c768-f7c0-4af1-af14-56c55831c4f1" containerID="1fdb8cab42a5776859d70d8bb130fe9d5482570e0ac6aa93f3ea1faae5301dfa" exitCode=0 Jul 25 02:05:45.256431 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:45.256306 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" Jul 25 02:05:45.256431 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:45.256327 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" event={"ID":"4766c768-f7c0-4af1-af14-56c55831c4f1","Type":"ContainerDied","Data":"1fdb8cab42a5776859d70d8bb130fe9d5482570e0ac6aa93f3ea1faae5301dfa"} Jul 25 02:05:45.256431 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:45.256369 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd" event={"ID":"4766c768-f7c0-4af1-af14-56c55831c4f1","Type":"ContainerDied","Data":"f3c6324370e440cb5cada8197a426682a594f4f13e226dfb8cd0f9d8967ae868"} Jul 25 02:05:45.256431 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:45.256390 2554 scope.go:117] "RemoveContainer" containerID="1fdb8cab42a5776859d70d8bb130fe9d5482570e0ac6aa93f3ea1faae5301dfa" Jul 25 02:05:45.265982 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:45.265967 2554 scope.go:117] "RemoveContainer" containerID="1fdb8cab42a5776859d70d8bb130fe9d5482570e0ac6aa93f3ea1faae5301dfa" Jul 25 02:05:45.266280 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:05:45.266260 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fdb8cab42a5776859d70d8bb130fe9d5482570e0ac6aa93f3ea1faae5301dfa\": container with ID starting with 1fdb8cab42a5776859d70d8bb130fe9d5482570e0ac6aa93f3ea1faae5301dfa not found: ID does not exist" containerID="1fdb8cab42a5776859d70d8bb130fe9d5482570e0ac6aa93f3ea1faae5301dfa" Jul 25 02:05:45.266348 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:45.266290 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fdb8cab42a5776859d70d8bb130fe9d5482570e0ac6aa93f3ea1faae5301dfa"} err="failed to get container status \"1fdb8cab42a5776859d70d8bb130fe9d5482570e0ac6aa93f3ea1faae5301dfa\": rpc error: code = NotFound desc = could not find container \"1fdb8cab42a5776859d70d8bb130fe9d5482570e0ac6aa93f3ea1faae5301dfa\": container with ID starting with 1fdb8cab42a5776859d70d8bb130fe9d5482570e0ac6aa93f3ea1faae5301dfa not found: ID does not exist" Jul 25 02:05:45.275948 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:45.275927 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd"] Jul 25 02:05:45.282483 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:45.282463 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c13f56-5d5f4f8f55-5rcqd"] Jul 25 02:05:45.382312 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:05:45.382240 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4766c768-f7c0-4af1-af14-56c55831c4f1" path="/var/lib/kubelet/pods/4766c768-f7c0-4af1-af14-56c55831c4f1/volumes" Jul 25 02:06:06.705372 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:06.705336 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px"] Jul 25 02:06:06.705953 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:06.705607 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" podUID="0163bfc9-2ca0-43bf-9d06-1f56db5bc728" containerName="maas-api" containerID="cri-o://be77c77846949f6c09ce2b7eded511baf11de1d35a985959f9c72c1a6c8e7fe7" gracePeriod=30 Jul 25 02:06:06.725710 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:06.725662 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" podUID="0163bfc9-2ca0-43bf-9d06-1f56db5bc728" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.65:8443/health\": dial tcp 10.133.0.65:8443: connect: connection refused" Jul 25 02:06:06.758996 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:06.758966 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv"] Jul 25 02:06:06.759225 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:06.759182 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv" podUID="75573941-78c1-4af4-b33f-83f31cbcde66" containerName="payload-processing" containerID="cri-o://6c3ec906d2555049604135f9da9d6dc05922b676c544d128026f93b09240af0b" gracePeriod=30 Jul 25 02:06:06.764030 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:06.764006 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm"] Jul 25 02:06:06.764214 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:06.764193 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm" podUID="76baa294-0c16-408c-b9a8-cd44e621ca1f" containerName="payload-pre-processing" containerID="cri-o://ac708ba81d8a15bd2c331669f956e9271431455872247dbfd938a2a715a50fdd" gracePeriod=30 Jul 25 02:06:07.039999 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.039968 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" Jul 25 02:06:07.043063 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.043042 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv" Jul 25 02:06:07.061004 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.060979 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bf2hq\" (UniqueName: \"kubernetes.io/projected/75573941-78c1-4af4-b33f-83f31cbcde66-kube-api-access-bf2hq\") pod \"75573941-78c1-4af4-b33f-83f31cbcde66\" (UID: \"75573941-78c1-4af4-b33f-83f31cbcde66\") " Jul 25 02:06:07.061118 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.061040 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-maas-api-tls\") pod \"0163bfc9-2ca0-43bf-9d06-1f56db5bc728\" (UID: \"0163bfc9-2ca0-43bf-9d06-1f56db5bc728\") " Jul 25 02:06:07.061118 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.061075 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjdq6\" (UniqueName: \"kubernetes.io/projected/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-kube-api-access-sjdq6\") pod \"0163bfc9-2ca0-43bf-9d06-1f56db5bc728\" (UID: \"0163bfc9-2ca0-43bf-9d06-1f56db5bc728\") " Jul 25 02:06:07.061118 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.061102 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75573941-78c1-4af4-b33f-83f31cbcde66-plugins-config-volume\") pod \"75573941-78c1-4af4-b33f-83f31cbcde66\" (UID: \"75573941-78c1-4af4-b33f-83f31cbcde66\") " Jul 25 02:06:07.061293 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.061212 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-openshift-service-ca\") pod \"0163bfc9-2ca0-43bf-9d06-1f56db5bc728\" (UID: \"0163bfc9-2ca0-43bf-9d06-1f56db5bc728\") " Jul 25 02:06:07.062017 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.061760 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0163bfc9-2ca0-43bf-9d06-1f56db5bc728" (UID: "0163bfc9-2ca0-43bf-9d06-1f56db5bc728"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:06:07.062619 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.062577 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75573941-78c1-4af4-b33f-83f31cbcde66-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "75573941-78c1-4af4-b33f-83f31cbcde66" (UID: "75573941-78c1-4af4-b33f-83f31cbcde66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:06:07.064112 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.064088 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0163bfc9-2ca0-43bf-9d06-1f56db5bc728" (UID: "0163bfc9-2ca0-43bf-9d06-1f56db5bc728"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 02:06:07.064205 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.064184 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-kube-api-access-sjdq6" (OuterVolumeSpecName: "kube-api-access-sjdq6") pod "0163bfc9-2ca0-43bf-9d06-1f56db5bc728" (UID: "0163bfc9-2ca0-43bf-9d06-1f56db5bc728"). InnerVolumeSpecName "kube-api-access-sjdq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:06:07.064251 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.064215 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75573941-78c1-4af4-b33f-83f31cbcde66-kube-api-access-bf2hq" (OuterVolumeSpecName: "kube-api-access-bf2hq") pod "75573941-78c1-4af4-b33f-83f31cbcde66" (UID: "75573941-78c1-4af4-b33f-83f31cbcde66"). InnerVolumeSpecName "kube-api-access-bf2hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:06:07.066401 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.066378 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm" Jul 25 02:06:07.162441 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.162416 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76baa294-0c16-408c-b9a8-cd44e621ca1f-plugins-config-volume\") pod \"76baa294-0c16-408c-b9a8-cd44e621ca1f\" (UID: \"76baa294-0c16-408c-b9a8-cd44e621ca1f\") " Jul 25 02:06:07.162565 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.162514 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd7cs\" (UniqueName: \"kubernetes.io/projected/76baa294-0c16-408c-b9a8-cd44e621ca1f-kube-api-access-bd7cs\") pod \"76baa294-0c16-408c-b9a8-cd44e621ca1f\" (UID: \"76baa294-0c16-408c-b9a8-cd44e621ca1f\") " Jul 25 02:06:07.162683 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.162671 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-openshift-service-ca\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:06:07.162732 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.162686 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf2hq\" (UniqueName: \"kubernetes.io/projected/75573941-78c1-4af4-b33f-83f31cbcde66-kube-api-access-bf2hq\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:06:07.162732 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.162696 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-maas-api-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:06:07.162732 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.162704 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjdq6\" (UniqueName: \"kubernetes.io/projected/0163bfc9-2ca0-43bf-9d06-1f56db5bc728-kube-api-access-sjdq6\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:06:07.162732 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.162712 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75573941-78c1-4af4-b33f-83f31cbcde66-plugins-config-volume\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:06:07.162887 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.162832 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76baa294-0c16-408c-b9a8-cd44e621ca1f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "76baa294-0c16-408c-b9a8-cd44e621ca1f" (UID: "76baa294-0c16-408c-b9a8-cd44e621ca1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:06:07.164584 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.164566 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76baa294-0c16-408c-b9a8-cd44e621ca1f-kube-api-access-bd7cs" (OuterVolumeSpecName: "kube-api-access-bd7cs") pod "76baa294-0c16-408c-b9a8-cd44e621ca1f" (UID: "76baa294-0c16-408c-b9a8-cd44e621ca1f"). InnerVolumeSpecName "kube-api-access-bd7cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:06:07.263337 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.263274 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd7cs\" (UniqueName: \"kubernetes.io/projected/76baa294-0c16-408c-b9a8-cd44e621ca1f-kube-api-access-bd7cs\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:06:07.263337 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.263295 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76baa294-0c16-408c-b9a8-cd44e621ca1f-plugins-config-volume\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:06:07.382375 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.382345 2554 generic.go:356] "Generic (PLEG): container finished" podID="0163bfc9-2ca0-43bf-9d06-1f56db5bc728" containerID="be77c77846949f6c09ce2b7eded511baf11de1d35a985959f9c72c1a6c8e7fe7" exitCode=0 Jul 25 02:06:07.382545 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.382458 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" Jul 25 02:06:07.382545 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.382354 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" event={"ID":"0163bfc9-2ca0-43bf-9d06-1f56db5bc728","Type":"ContainerDied","Data":"be77c77846949f6c09ce2b7eded511baf11de1d35a985959f9c72c1a6c8e7fe7"} Jul 25 02:06:07.382545 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.382535 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px" event={"ID":"0163bfc9-2ca0-43bf-9d06-1f56db5bc728","Type":"ContainerDied","Data":"9220d162ad5a49a99b0af21edce15d4bd0da186817675f2740141bd8f3fda6cb"} Jul 25 02:06:07.382715 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.382575 2554 scope.go:117] "RemoveContainer" containerID="be77c77846949f6c09ce2b7eded511baf11de1d35a985959f9c72c1a6c8e7fe7" Jul 25 02:06:07.384341 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.384240 2554 generic.go:356] "Generic (PLEG): container finished" podID="75573941-78c1-4af4-b33f-83f31cbcde66" containerID="6c3ec906d2555049604135f9da9d6dc05922b676c544d128026f93b09240af0b" exitCode=0 Jul 25 02:06:07.384341 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.384272 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv" event={"ID":"75573941-78c1-4af4-b33f-83f31cbcde66","Type":"ContainerDied","Data":"6c3ec906d2555049604135f9da9d6dc05922b676c544d128026f93b09240af0b"} Jul 25 02:06:07.384341 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.384334 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv" event={"ID":"75573941-78c1-4af4-b33f-83f31cbcde66","Type":"ContainerDied","Data":"a9fc89a9ddd8324ef96720988f4a0e0fef19dcd0a56ba30668037496955815c7"} Jul 25 02:06:07.384580 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.384283 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv" Jul 25 02:06:07.386537 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.386121 2554 generic.go:356] "Generic (PLEG): container finished" podID="76baa294-0c16-408c-b9a8-cd44e621ca1f" containerID="ac708ba81d8a15bd2c331669f956e9271431455872247dbfd938a2a715a50fdd" exitCode=0 Jul 25 02:06:07.386537 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.386165 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm" event={"ID":"76baa294-0c16-408c-b9a8-cd44e621ca1f","Type":"ContainerDied","Data":"ac708ba81d8a15bd2c331669f956e9271431455872247dbfd938a2a715a50fdd"} Jul 25 02:06:07.386537 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.386204 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm" event={"ID":"76baa294-0c16-408c-b9a8-cd44e621ca1f","Type":"ContainerDied","Data":"1aedae3b5f974d5e32c6d2b27a8c4e02dc19991422b9b74f5af7d333375bfb16"} Jul 25 02:06:07.386537 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.386261 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm" Jul 25 02:06:07.395024 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.395009 2554 scope.go:117] "RemoveContainer" containerID="be77c77846949f6c09ce2b7eded511baf11de1d35a985959f9c72c1a6c8e7fe7" Jul 25 02:06:07.395276 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:06:07.395257 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be77c77846949f6c09ce2b7eded511baf11de1d35a985959f9c72c1a6c8e7fe7\": container with ID starting with be77c77846949f6c09ce2b7eded511baf11de1d35a985959f9c72c1a6c8e7fe7 not found: ID does not exist" containerID="be77c77846949f6c09ce2b7eded511baf11de1d35a985959f9c72c1a6c8e7fe7" Jul 25 02:06:07.395362 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.395283 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be77c77846949f6c09ce2b7eded511baf11de1d35a985959f9c72c1a6c8e7fe7"} err="failed to get container status \"be77c77846949f6c09ce2b7eded511baf11de1d35a985959f9c72c1a6c8e7fe7\": rpc error: code = NotFound desc = could not find container \"be77c77846949f6c09ce2b7eded511baf11de1d35a985959f9c72c1a6c8e7fe7\": container with ID starting with be77c77846949f6c09ce2b7eded511baf11de1d35a985959f9c72c1a6c8e7fe7 not found: ID does not exist" Jul 25 02:06:07.395362 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.395299 2554 scope.go:117] "RemoveContainer" containerID="6c3ec906d2555049604135f9da9d6dc05922b676c544d128026f93b09240af0b" Jul 25 02:06:07.404984 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.404970 2554 scope.go:117] "RemoveContainer" containerID="6c3ec906d2555049604135f9da9d6dc05922b676c544d128026f93b09240af0b" Jul 25 02:06:07.405215 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:06:07.405199 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3ec906d2555049604135f9da9d6dc05922b676c544d128026f93b09240af0b\": container with ID starting with 6c3ec906d2555049604135f9da9d6dc05922b676c544d128026f93b09240af0b not found: ID does not exist" containerID="6c3ec906d2555049604135f9da9d6dc05922b676c544d128026f93b09240af0b" Jul 25 02:06:07.405261 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.405222 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3ec906d2555049604135f9da9d6dc05922b676c544d128026f93b09240af0b"} err="failed to get container status \"6c3ec906d2555049604135f9da9d6dc05922b676c544d128026f93b09240af0b\": rpc error: code = NotFound desc = could not find container \"6c3ec906d2555049604135f9da9d6dc05922b676c544d128026f93b09240af0b\": container with ID starting with 6c3ec906d2555049604135f9da9d6dc05922b676c544d128026f93b09240af0b not found: ID does not exist" Jul 25 02:06:07.405261 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.405238 2554 scope.go:117] "RemoveContainer" containerID="ac708ba81d8a15bd2c331669f956e9271431455872247dbfd938a2a715a50fdd" Jul 25 02:06:07.412790 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.412771 2554 scope.go:117] "RemoveContainer" containerID="ac708ba81d8a15bd2c331669f956e9271431455872247dbfd938a2a715a50fdd" Jul 25 02:06:07.413017 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:06:07.412998 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac708ba81d8a15bd2c331669f956e9271431455872247dbfd938a2a715a50fdd\": container with ID starting with ac708ba81d8a15bd2c331669f956e9271431455872247dbfd938a2a715a50fdd not found: ID does not exist" containerID="ac708ba81d8a15bd2c331669f956e9271431455872247dbfd938a2a715a50fdd" Jul 25 02:06:07.413082 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.413022 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac708ba81d8a15bd2c331669f956e9271431455872247dbfd938a2a715a50fdd"} err="failed to get container status \"ac708ba81d8a15bd2c331669f956e9271431455872247dbfd938a2a715a50fdd\": rpc error: code = NotFound desc = could not find container \"ac708ba81d8a15bd2c331669f956e9271431455872247dbfd938a2a715a50fdd\": container with ID starting with ac708ba81d8a15bd2c331669f956e9271431455872247dbfd938a2a715a50fdd not found: ID does not exist" Jul 25 02:06:07.413953 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.413933 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px"] Jul 25 02:06:07.417287 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.417267 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a3abd-5cdc468cc8-s82px"] Jul 25 02:06:07.426316 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.426297 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv"] Jul 25 02:06:07.428381 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.428365 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1a3abd-656f556b84-5mchv"] Jul 25 02:06:07.436782 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.436765 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm"] Jul 25 02:06:07.438709 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:07.438691 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1a3abd-68bf989cf6-74tsm"] Jul 25 02:06:09.381111 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:09.381081 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0163bfc9-2ca0-43bf-9d06-1f56db5bc728" path="/var/lib/kubelet/pods/0163bfc9-2ca0-43bf-9d06-1f56db5bc728/volumes" Jul 25 02:06:09.381490 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:09.381396 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75573941-78c1-4af4-b33f-83f31cbcde66" path="/var/lib/kubelet/pods/75573941-78c1-4af4-b33f-83f31cbcde66/volumes" Jul 25 02:06:09.381699 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:09.381682 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76baa294-0c16-408c-b9a8-cd44e621ca1f" path="/var/lib/kubelet/pods/76baa294-0c16-408c-b9a8-cd44e621ca1f/volumes" Jul 25 02:06:21.441261 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.441173 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc"] Jul 25 02:06:21.441873 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.441725 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4766c768-f7c0-4af1-af14-56c55831c4f1" containerName="maas-api" Jul 25 02:06:21.441873 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.441746 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4766c768-f7c0-4af1-af14-56c55831c4f1" containerName="maas-api" Jul 25 02:06:21.441873 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.441760 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0163bfc9-2ca0-43bf-9d06-1f56db5bc728" containerName="maas-api" Jul 25 02:06:21.441873 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.441771 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0163bfc9-2ca0-43bf-9d06-1f56db5bc728" containerName="maas-api" Jul 25 02:06:21.441873 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.441789 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76baa294-0c16-408c-b9a8-cd44e621ca1f" containerName="payload-pre-processing" Jul 25 02:06:21.441873 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.441822 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="76baa294-0c16-408c-b9a8-cd44e621ca1f" containerName="payload-pre-processing" Jul 25 02:06:21.441873 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.441858 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75573941-78c1-4af4-b33f-83f31cbcde66" containerName="payload-processing" Jul 25 02:06:21.441873 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.441867 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="75573941-78c1-4af4-b33f-83f31cbcde66" containerName="payload-processing" Jul 25 02:06:21.442192 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.441969 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="76baa294-0c16-408c-b9a8-cd44e621ca1f" containerName="payload-pre-processing" Jul 25 02:06:21.442192 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.441984 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4766c768-f7c0-4af1-af14-56c55831c4f1" containerName="maas-api" Jul 25 02:06:21.442192 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.441994 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0163bfc9-2ca0-43bf-9d06-1f56db5bc728" containerName="maas-api" Jul 25 02:06:21.442192 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.442003 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="75573941-78c1-4af4-b33f-83f31cbcde66" containerName="payload-processing" Jul 25 02:06:21.446394 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.445451 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" Jul 25 02:06:21.448122 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.448105 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a9b5fd-serving-cert\"" Jul 25 02:06:21.455129 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.454783 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc"] Jul 25 02:06:21.581299 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.581263 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-maas-api-tls\") pod \"maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc\" (UID: \"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" Jul 25 02:06:21.581299 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.581298 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2mwk\" (UniqueName: \"kubernetes.io/projected/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-kube-api-access-p2mwk\") pod \"maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc\" (UID: \"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" Jul 25 02:06:21.581516 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.581384 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-openshift-service-ca\") pod \"maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc\" (UID: \"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" Jul 25 02:06:21.682399 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.682370 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-maas-api-tls\") pod \"maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc\" (UID: \"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" Jul 25 02:06:21.682583 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.682405 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2mwk\" (UniqueName: \"kubernetes.io/projected/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-kube-api-access-p2mwk\") pod \"maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc\" (UID: \"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" Jul 25 02:06:21.682583 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.682428 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-openshift-service-ca\") pod \"maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc\" (UID: \"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" Jul 25 02:06:21.682583 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:06:21.682534 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-serving-cert: secret "maas-api-e2e-api-a-a9b5fd-serving-cert" not found Jul 25 02:06:21.682759 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:06:21.682645 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-maas-api-tls podName:1a74be98-bb0e-49fe-a5ac-24e876e4ec5e nodeName:}" failed. No retries permitted until 2026-07-25 02:06:22.182623197 +0000 UTC m=+3323.304109028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-maas-api-tls") pod "maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" (UID: "1a74be98-bb0e-49fe-a5ac-24e876e4ec5e") : secret "maas-api-e2e-api-a-a9b5fd-serving-cert" not found Jul 25 02:06:21.683165 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.683142 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-openshift-service-ca\") pod \"maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc\" (UID: \"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" Jul 25 02:06:21.692725 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:21.692673 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2mwk\" (UniqueName: \"kubernetes.io/projected/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-kube-api-access-p2mwk\") pod \"maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc\" (UID: \"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" Jul 25 02:06:22.118294 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.118215 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk"] Jul 25 02:06:22.121741 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.121721 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk" Jul 25 02:06:22.123758 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.123739 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a9b5fd\"" Jul 25 02:06:22.123895 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.123792 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a9b5fd-dockercfg-rlgsp\"" Jul 25 02:06:22.131868 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.131845 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk"] Jul 25 02:06:22.153998 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.153974 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8"] Jul 25 02:06:22.157380 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.157364 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8" Jul 25 02:06:22.167027 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.167003 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8"] Jul 25 02:06:22.188161 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.188139 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxls2\" (UniqueName: \"kubernetes.io/projected/ac1012ca-c394-4f5c-979a-0a84e99fe316-kube-api-access-xxls2\") pod \"payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk\" (UID: \"ac1012ca-c394-4f5c-979a-0a84e99fe316\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk" Jul 25 02:06:22.188291 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.188170 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1012ca-c394-4f5c-979a-0a84e99fe316-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk\" (UID: \"ac1012ca-c394-4f5c-979a-0a84e99fe316\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk" Jul 25 02:06:22.188291 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.188245 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-maas-api-tls\") pod \"maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc\" (UID: \"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" Jul 25 02:06:22.190833 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.190787 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-maas-api-tls\") pod \"maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc\" (UID: \"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" Jul 25 02:06:22.289174 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.289141 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh49p\" (UniqueName: \"kubernetes.io/projected/ae7cdbd1-7fb7-4bec-915d-8744fc00bc28-kube-api-access-zh49p\") pod \"payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8\" (UID: \"ae7cdbd1-7fb7-4bec-915d-8744fc00bc28\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8" Jul 25 02:06:22.289348 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.289196 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxls2\" (UniqueName: \"kubernetes.io/projected/ac1012ca-c394-4f5c-979a-0a84e99fe316-kube-api-access-xxls2\") pod \"payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk\" (UID: \"ac1012ca-c394-4f5c-979a-0a84e99fe316\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk" Jul 25 02:06:22.289348 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.289216 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1012ca-c394-4f5c-979a-0a84e99fe316-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk\" (UID: \"ac1012ca-c394-4f5c-979a-0a84e99fe316\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk" Jul 25 02:06:22.289348 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.289304 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae7cdbd1-7fb7-4bec-915d-8744fc00bc28-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8\" (UID: \"ae7cdbd1-7fb7-4bec-915d-8744fc00bc28\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8" Jul 25 02:06:22.290099 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.290077 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1012ca-c394-4f5c-979a-0a84e99fe316-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk\" (UID: \"ac1012ca-c394-4f5c-979a-0a84e99fe316\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk" Jul 25 02:06:22.296901 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.296873 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxls2\" (UniqueName: \"kubernetes.io/projected/ac1012ca-c394-4f5c-979a-0a84e99fe316-kube-api-access-xxls2\") pod \"payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk\" (UID: \"ac1012ca-c394-4f5c-979a-0a84e99fe316\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk" Jul 25 02:06:22.359300 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.359270 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" Jul 25 02:06:22.389912 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.389883 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae7cdbd1-7fb7-4bec-915d-8744fc00bc28-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8\" (UID: \"ae7cdbd1-7fb7-4bec-915d-8744fc00bc28\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8" Jul 25 02:06:22.390068 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.389973 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh49p\" (UniqueName: \"kubernetes.io/projected/ae7cdbd1-7fb7-4bec-915d-8744fc00bc28-kube-api-access-zh49p\") pod \"payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8\" (UID: \"ae7cdbd1-7fb7-4bec-915d-8744fc00bc28\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8" Jul 25 02:06:22.390884 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.390862 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae7cdbd1-7fb7-4bec-915d-8744fc00bc28-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8\" (UID: \"ae7cdbd1-7fb7-4bec-915d-8744fc00bc28\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8" Jul 25 02:06:22.397206 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.397183 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh49p\" (UniqueName: \"kubernetes.io/projected/ae7cdbd1-7fb7-4bec-915d-8744fc00bc28-kube-api-access-zh49p\") pod \"payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8\" (UID: \"ae7cdbd1-7fb7-4bec-915d-8744fc00bc28\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8" Jul 25 02:06:22.434047 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.434018 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk" Jul 25 02:06:22.468164 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.468129 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8" Jul 25 02:06:22.509462 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.509424 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc"] Jul 25 02:06:22.514194 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.514171 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 25 02:06:22.598322 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.598243 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk"] Jul 25 02:06:22.602367 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:06:22.602327 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1012ca_c394_4f5c_979a_0a84e99fe316.slice/crio-d01c0ded9be560f2923ddfc09621355814e082cf5baa0f8f29fc69afcd29472d WatchSource:0}: Error finding container d01c0ded9be560f2923ddfc09621355814e082cf5baa0f8f29fc69afcd29472d: Status 404 returned error can't find the container with id d01c0ded9be560f2923ddfc09621355814e082cf5baa0f8f29fc69afcd29472d Jul 25 02:06:22.636976 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:22.636949 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8"] Jul 25 02:06:22.638212 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:06:22.638187 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7cdbd1_7fb7_4bec_915d_8744fc00bc28.slice/crio-aeb3373b504b1a6f4c8cbc278bb79703b8756f55d5b576e7800d01d68a2bd10f WatchSource:0}: Error finding container aeb3373b504b1a6f4c8cbc278bb79703b8756f55d5b576e7800d01d68a2bd10f: Status 404 returned error can't find the container with id aeb3373b504b1a6f4c8cbc278bb79703b8756f55d5b576e7800d01d68a2bd10f Jul 25 02:06:23.478780 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:23.478747 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" event={"ID":"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e","Type":"ContainerStarted","Data":"da2f0ff51c5c51a8bcd188b4185b36716afd0dd6d2a664241265e52c928f9d4e"} Jul 25 02:06:23.479458 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:23.478844 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" Jul 25 02:06:23.479458 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:23.478870 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" event={"ID":"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e","Type":"ContainerStarted","Data":"cd62a4e6c64aab5f02b691d1e14e1619562f99b388acf06dbfb1790820855217"} Jul 25 02:06:23.481081 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:23.481050 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk" event={"ID":"ac1012ca-c394-4f5c-979a-0a84e99fe316","Type":"ContainerStarted","Data":"d4c40f8c3d7cab6443fda06168ca042765d7f388befbb1fd133a49d31765a0c9"} Jul 25 02:06:23.481340 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:23.481301 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk" Jul 25 02:06:23.481499 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:23.481482 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk" event={"ID":"ac1012ca-c394-4f5c-979a-0a84e99fe316","Type":"ContainerStarted","Data":"d01c0ded9be560f2923ddfc09621355814e082cf5baa0f8f29fc69afcd29472d"} Jul 25 02:06:23.483112 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:23.483090 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8" event={"ID":"ae7cdbd1-7fb7-4bec-915d-8744fc00bc28","Type":"ContainerStarted","Data":"5482912963bdf549debb17c8ce87b7ede5ecc433104c796beb102f9a54a6c91b"} Jul 25 02:06:23.483216 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:23.483135 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8" event={"ID":"ae7cdbd1-7fb7-4bec-915d-8744fc00bc28","Type":"ContainerStarted","Data":"aeb3373b504b1a6f4c8cbc278bb79703b8756f55d5b576e7800d01d68a2bd10f"} Jul 25 02:06:23.483216 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:23.483164 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8" Jul 25 02:06:23.497435 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:23.497394 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" podStartSLOduration=1.936784423 podStartE2EDuration="2.497379705s" totalImagesPullingTime="560.595282ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:06:23.494429391 +0000 UTC m=+3324.615915242" watchObservedRunningTime="2026-07-25 02:06:23.497379705 +0000 UTC m=+3324.618865554" Jul 25 02:06:23.515890 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:23.515841 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk" podStartSLOduration=1.5158280849999999 podStartE2EDuration="1.515828085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:06:23.513243403 +0000 UTC m=+3324.634729276" watchObservedRunningTime="2026-07-25 02:06:23.515828085 +0000 UTC m=+3324.637313935" Jul 25 02:06:23.528775 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:23.528733 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8" podStartSLOduration=1.52871999 podStartE2EDuration="1.52871999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:06:23.526931942 +0000 UTC m=+3324.648417795" watchObservedRunningTime="2026-07-25 02:06:23.52871999 +0000 UTC m=+3324.650205840" Jul 25 02:06:29.493931 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:29.493892 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" Jul 25 02:06:29.624972 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:29.624938 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6"] Jul 25 02:06:29.630156 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:29.630135 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" Jul 25 02:06:29.632262 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:29.632239 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6fd1ef-serving-cert\"" Jul 25 02:06:29.650610 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:29.650587 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6"] Jul 25 02:06:29.756300 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:29.756226 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8712ad-abee-4f21-b824-b9936c2b8c82-openshift-service-ca\") pod \"maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6\" (UID: \"4b8712ad-abee-4f21-b824-b9936c2b8c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" Jul 25 02:06:29.756300 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:29.756257 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbskn\" (UniqueName: \"kubernetes.io/projected/4b8712ad-abee-4f21-b824-b9936c2b8c82-kube-api-access-sbskn\") pod \"maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6\" (UID: \"4b8712ad-abee-4f21-b824-b9936c2b8c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" Jul 25 02:06:29.756300 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:29.756284 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8712ad-abee-4f21-b824-b9936c2b8c82-maas-api-tls\") pod \"maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6\" (UID: \"4b8712ad-abee-4f21-b824-b9936c2b8c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" Jul 25 02:06:29.857599 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:29.857572 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8712ad-abee-4f21-b824-b9936c2b8c82-openshift-service-ca\") pod \"maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6\" (UID: \"4b8712ad-abee-4f21-b824-b9936c2b8c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" Jul 25 02:06:29.857734 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:29.857600 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbskn\" (UniqueName: \"kubernetes.io/projected/4b8712ad-abee-4f21-b824-b9936c2b8c82-kube-api-access-sbskn\") pod \"maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6\" (UID: \"4b8712ad-abee-4f21-b824-b9936c2b8c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" Jul 25 02:06:29.857734 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:29.857627 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8712ad-abee-4f21-b824-b9936c2b8c82-maas-api-tls\") pod \"maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6\" (UID: \"4b8712ad-abee-4f21-b824-b9936c2b8c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" Jul 25 02:06:29.857734 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:06:29.857712 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-serving-cert: secret "maas-api-e2e-api-b-6fd1ef-serving-cert" not found Jul 25 02:06:29.857942 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:06:29.857772 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b8712ad-abee-4f21-b824-b9936c2b8c82-maas-api-tls podName:4b8712ad-abee-4f21-b824-b9936c2b8c82 nodeName:}" failed. No retries permitted until 2026-07-25 02:06:30.357756379 +0000 UTC m=+3331.479242207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b8712ad-abee-4f21-b824-b9936c2b8c82-maas-api-tls") pod "maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" (UID: "4b8712ad-abee-4f21-b824-b9936c2b8c82") : secret "maas-api-e2e-api-b-6fd1ef-serving-cert" not found Jul 25 02:06:29.858335 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:29.858311 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8712ad-abee-4f21-b824-b9936c2b8c82-openshift-service-ca\") pod \"maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6\" (UID: \"4b8712ad-abee-4f21-b824-b9936c2b8c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" Jul 25 02:06:29.866256 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:29.866235 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbskn\" (UniqueName: \"kubernetes.io/projected/4b8712ad-abee-4f21-b824-b9936c2b8c82-kube-api-access-sbskn\") pod \"maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6\" (UID: \"4b8712ad-abee-4f21-b824-b9936c2b8c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" Jul 25 02:06:30.361739 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:30.361706 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8712ad-abee-4f21-b824-b9936c2b8c82-maas-api-tls\") pod \"maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6\" (UID: \"4b8712ad-abee-4f21-b824-b9936c2b8c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" Jul 25 02:06:30.364314 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:30.364293 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8712ad-abee-4f21-b824-b9936c2b8c82-maas-api-tls\") pod \"maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6\" (UID: \"4b8712ad-abee-4f21-b824-b9936c2b8c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" Jul 25 02:06:30.542379 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:30.542346 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" Jul 25 02:06:30.692735 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:30.692703 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6"] Jul 25 02:06:30.693443 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:06:30.693416 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8712ad_abee_4f21_b824_b9936c2b8c82.slice/crio-2c9eb856391b5cbff4cd0b01db7555c16dc03269d024ba79c829655e8a8fb2dc WatchSource:0}: Error finding container 2c9eb856391b5cbff4cd0b01db7555c16dc03269d024ba79c829655e8a8fb2dc: Status 404 returned error can't find the container with id 2c9eb856391b5cbff4cd0b01db7555c16dc03269d024ba79c829655e8a8fb2dc Jul 25 02:06:31.531309 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:31.531275 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" event={"ID":"4b8712ad-abee-4f21-b824-b9936c2b8c82","Type":"ContainerStarted","Data":"af8c08c0b185498432038427b44b29464f0a53a60cc7f3313a3e3617f0b08eed"} Jul 25 02:06:31.531497 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:31.531328 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" event={"ID":"4b8712ad-abee-4f21-b824-b9936c2b8c82","Type":"ContainerStarted","Data":"2c9eb856391b5cbff4cd0b01db7555c16dc03269d024ba79c829655e8a8fb2dc"} Jul 25 02:06:31.531497 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:31.531412 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" Jul 25 02:06:31.551495 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:31.551445 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" podStartSLOduration=2.122637798 podStartE2EDuration="2.551431749s" totalImagesPullingTime="428.793951ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:06:31.550113233 +0000 UTC m=+3332.671599084" watchObservedRunningTime="2026-07-25 02:06:31.551431749 +0000 UTC m=+3332.672917598" Jul 25 02:06:34.491707 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:34.491680 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8" Jul 25 02:06:35.498664 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:35.498636 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk" Jul 25 02:06:37.542163 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:37.542134 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" Jul 25 02:06:56.944991 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:56.944954 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc"] Jul 25 02:06:56.945535 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:56.945259 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" podUID="1a74be98-bb0e-49fe-a5ac-24e876e4ec5e" containerName="maas-api" containerID="cri-o://da2f0ff51c5c51a8bcd188b4185b36716afd0dd6d2a664241265e52c928f9d4e" gracePeriod=30 Jul 25 02:06:57.009726 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.009688 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8"] Jul 25 02:06:57.010015 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.009977 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8" podUID="ae7cdbd1-7fb7-4bec-915d-8744fc00bc28" containerName="payload-processing" containerID="cri-o://5482912963bdf549debb17c8ce87b7ede5ecc433104c796beb102f9a54a6c91b" gracePeriod=30 Jul 25 02:06:57.017277 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.017251 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk"] Jul 25 02:06:57.017617 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.017458 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk" podUID="ac1012ca-c394-4f5c-979a-0a84e99fe316" containerName="payload-pre-processing" containerID="cri-o://d4c40f8c3d7cab6443fda06168ca042765d7f388befbb1fd133a49d31765a0c9" gracePeriod=30 Jul 25 02:06:57.312723 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.312700 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8" Jul 25 02:06:57.315881 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.315859 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" Jul 25 02:06:57.319126 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.319108 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk" Jul 25 02:06:57.399129 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.399103 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxls2\" (UniqueName: \"kubernetes.io/projected/ac1012ca-c394-4f5c-979a-0a84e99fe316-kube-api-access-xxls2\") pod \"ac1012ca-c394-4f5c-979a-0a84e99fe316\" (UID: \"ac1012ca-c394-4f5c-979a-0a84e99fe316\") " Jul 25 02:06:57.399129 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.399133 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh49p\" (UniqueName: \"kubernetes.io/projected/ae7cdbd1-7fb7-4bec-915d-8744fc00bc28-kube-api-access-zh49p\") pod \"ae7cdbd1-7fb7-4bec-915d-8744fc00bc28\" (UID: \"ae7cdbd1-7fb7-4bec-915d-8744fc00bc28\") " Jul 25 02:06:57.399377 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.399160 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-maas-api-tls\") pod \"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e\" (UID: \"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e\") " Jul 25 02:06:57.399377 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.399184 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-openshift-service-ca\") pod \"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e\" (UID: \"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e\") " Jul 25 02:06:57.399377 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.399253 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae7cdbd1-7fb7-4bec-915d-8744fc00bc28-plugins-config-volume\") pod \"ae7cdbd1-7fb7-4bec-915d-8744fc00bc28\" (UID: \"ae7cdbd1-7fb7-4bec-915d-8744fc00bc28\") " Jul 25 02:06:57.399377 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.399297 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2mwk\" (UniqueName: \"kubernetes.io/projected/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-kube-api-access-p2mwk\") pod \"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e\" (UID: \"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e\") " Jul 25 02:06:57.399377 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.399354 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1012ca-c394-4f5c-979a-0a84e99fe316-plugins-config-volume\") pod \"ac1012ca-c394-4f5c-979a-0a84e99fe316\" (UID: \"ac1012ca-c394-4f5c-979a-0a84e99fe316\") " Jul 25 02:06:57.400141 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.399668 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1a74be98-bb0e-49fe-a5ac-24e876e4ec5e" (UID: "1a74be98-bb0e-49fe-a5ac-24e876e4ec5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:06:57.400141 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.399689 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae7cdbd1-7fb7-4bec-915d-8744fc00bc28-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae7cdbd1-7fb7-4bec-915d-8744fc00bc28" (UID: "ae7cdbd1-7fb7-4bec-915d-8744fc00bc28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:06:57.400141 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.399765 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac1012ca-c394-4f5c-979a-0a84e99fe316-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac1012ca-c394-4f5c-979a-0a84e99fe316" (UID: "ac1012ca-c394-4f5c-979a-0a84e99fe316"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:06:57.401775 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.401748 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7cdbd1-7fb7-4bec-915d-8744fc00bc28-kube-api-access-zh49p" (OuterVolumeSpecName: "kube-api-access-zh49p") pod "ae7cdbd1-7fb7-4bec-915d-8744fc00bc28" (UID: "ae7cdbd1-7fb7-4bec-915d-8744fc00bc28"). InnerVolumeSpecName "kube-api-access-zh49p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:06:57.402019 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.401994 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1012ca-c394-4f5c-979a-0a84e99fe316-kube-api-access-xxls2" (OuterVolumeSpecName: "kube-api-access-xxls2") pod "ac1012ca-c394-4f5c-979a-0a84e99fe316" (UID: "ac1012ca-c394-4f5c-979a-0a84e99fe316"). InnerVolumeSpecName "kube-api-access-xxls2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:06:57.402127 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.402031 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1a74be98-bb0e-49fe-a5ac-24e876e4ec5e" (UID: "1a74be98-bb0e-49fe-a5ac-24e876e4ec5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 02:06:57.402127 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.402064 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-kube-api-access-p2mwk" (OuterVolumeSpecName: "kube-api-access-p2mwk") pod "1a74be98-bb0e-49fe-a5ac-24e876e4ec5e" (UID: "1a74be98-bb0e-49fe-a5ac-24e876e4ec5e"). InnerVolumeSpecName "kube-api-access-p2mwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:06:57.500687 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.500612 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae7cdbd1-7fb7-4bec-915d-8744fc00bc28-plugins-config-volume\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:06:57.500687 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.500641 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2mwk\" (UniqueName: \"kubernetes.io/projected/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-kube-api-access-p2mwk\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:06:57.500687 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.500650 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1012ca-c394-4f5c-979a-0a84e99fe316-plugins-config-volume\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:06:57.500687 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.500659 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxls2\" (UniqueName: \"kubernetes.io/projected/ac1012ca-c394-4f5c-979a-0a84e99fe316-kube-api-access-xxls2\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:06:57.500687 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.500668 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh49p\" (UniqueName: \"kubernetes.io/projected/ae7cdbd1-7fb7-4bec-915d-8744fc00bc28-kube-api-access-zh49p\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:06:57.500687 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.500681 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-maas-api-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:06:57.500687 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.500692 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e-openshift-service-ca\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:06:57.672042 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.672000 2554 generic.go:356] "Generic (PLEG): container finished" podID="1a74be98-bb0e-49fe-a5ac-24e876e4ec5e" containerID="da2f0ff51c5c51a8bcd188b4185b36716afd0dd6d2a664241265e52c928f9d4e" exitCode=0 Jul 25 02:06:57.672211 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.672057 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" Jul 25 02:06:57.672211 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.672047 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" event={"ID":"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e","Type":"ContainerDied","Data":"da2f0ff51c5c51a8bcd188b4185b36716afd0dd6d2a664241265e52c928f9d4e"} Jul 25 02:06:57.672211 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.672200 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc" event={"ID":"1a74be98-bb0e-49fe-a5ac-24e876e4ec5e","Type":"ContainerDied","Data":"cd62a4e6c64aab5f02b691d1e14e1619562f99b388acf06dbfb1790820855217"} Jul 25 02:06:57.672392 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.672231 2554 scope.go:117] "RemoveContainer" containerID="da2f0ff51c5c51a8bcd188b4185b36716afd0dd6d2a664241265e52c928f9d4e" Jul 25 02:06:57.674678 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.674513 2554 generic.go:356] "Generic (PLEG): container finished" podID="ac1012ca-c394-4f5c-979a-0a84e99fe316" containerID="d4c40f8c3d7cab6443fda06168ca042765d7f388befbb1fd133a49d31765a0c9" exitCode=0 Jul 25 02:06:57.674678 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.674578 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk" Jul 25 02:06:57.674678 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.674586 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk" event={"ID":"ac1012ca-c394-4f5c-979a-0a84e99fe316","Type":"ContainerDied","Data":"d4c40f8c3d7cab6443fda06168ca042765d7f388befbb1fd133a49d31765a0c9"} Jul 25 02:06:57.674678 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.674625 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk" event={"ID":"ac1012ca-c394-4f5c-979a-0a84e99fe316","Type":"ContainerDied","Data":"d01c0ded9be560f2923ddfc09621355814e082cf5baa0f8f29fc69afcd29472d"} Jul 25 02:06:57.677786 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.677667 2554 generic.go:356] "Generic (PLEG): container finished" podID="ae7cdbd1-7fb7-4bec-915d-8744fc00bc28" containerID="5482912963bdf549debb17c8ce87b7ede5ecc433104c796beb102f9a54a6c91b" exitCode=0 Jul 25 02:06:57.678049 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.677750 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8" Jul 25 02:06:57.678237 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.677770 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8" event={"ID":"ae7cdbd1-7fb7-4bec-915d-8744fc00bc28","Type":"ContainerDied","Data":"5482912963bdf549debb17c8ce87b7ede5ecc433104c796beb102f9a54a6c91b"} Jul 25 02:06:57.678343 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.678261 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8" event={"ID":"ae7cdbd1-7fb7-4bec-915d-8744fc00bc28","Type":"ContainerDied","Data":"aeb3373b504b1a6f4c8cbc278bb79703b8756f55d5b576e7800d01d68a2bd10f"} Jul 25 02:06:57.684062 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.684044 2554 scope.go:117] "RemoveContainer" containerID="da2f0ff51c5c51a8bcd188b4185b36716afd0dd6d2a664241265e52c928f9d4e" Jul 25 02:06:57.684365 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:06:57.684340 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da2f0ff51c5c51a8bcd188b4185b36716afd0dd6d2a664241265e52c928f9d4e\": container with ID starting with da2f0ff51c5c51a8bcd188b4185b36716afd0dd6d2a664241265e52c928f9d4e not found: ID does not exist" containerID="da2f0ff51c5c51a8bcd188b4185b36716afd0dd6d2a664241265e52c928f9d4e" Jul 25 02:06:57.684429 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.684378 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da2f0ff51c5c51a8bcd188b4185b36716afd0dd6d2a664241265e52c928f9d4e"} err="failed to get container status \"da2f0ff51c5c51a8bcd188b4185b36716afd0dd6d2a664241265e52c928f9d4e\": rpc error: code = NotFound desc = could not find container \"da2f0ff51c5c51a8bcd188b4185b36716afd0dd6d2a664241265e52c928f9d4e\": container with ID starting with da2f0ff51c5c51a8bcd188b4185b36716afd0dd6d2a664241265e52c928f9d4e not found: ID does not exist" Jul 25 02:06:57.684429 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.684401 2554 scope.go:117] "RemoveContainer" containerID="d4c40f8c3d7cab6443fda06168ca042765d7f388befbb1fd133a49d31765a0c9" Jul 25 02:06:57.692851 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.692832 2554 scope.go:117] "RemoveContainer" containerID="d4c40f8c3d7cab6443fda06168ca042765d7f388befbb1fd133a49d31765a0c9" Jul 25 02:06:57.693086 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:06:57.693068 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4c40f8c3d7cab6443fda06168ca042765d7f388befbb1fd133a49d31765a0c9\": container with ID starting with d4c40f8c3d7cab6443fda06168ca042765d7f388befbb1fd133a49d31765a0c9 not found: ID does not exist" containerID="d4c40f8c3d7cab6443fda06168ca042765d7f388befbb1fd133a49d31765a0c9" Jul 25 02:06:57.693157 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.693097 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4c40f8c3d7cab6443fda06168ca042765d7f388befbb1fd133a49d31765a0c9"} err="failed to get container status \"d4c40f8c3d7cab6443fda06168ca042765d7f388befbb1fd133a49d31765a0c9\": rpc error: code = NotFound desc = could not find container \"d4c40f8c3d7cab6443fda06168ca042765d7f388befbb1fd133a49d31765a0c9\": container with ID starting with d4c40f8c3d7cab6443fda06168ca042765d7f388befbb1fd133a49d31765a0c9 not found: ID does not exist" Jul 25 02:06:57.693157 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.693119 2554 scope.go:117] "RemoveContainer" containerID="5482912963bdf549debb17c8ce87b7ede5ecc433104c796beb102f9a54a6c91b" Jul 25 02:06:57.696237 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.696214 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc"] Jul 25 02:06:57.701156 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.701133 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a9b5fd-9b8d486cc-svrhc"] Jul 25 02:06:57.702556 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.702540 2554 scope.go:117] "RemoveContainer" containerID="5482912963bdf549debb17c8ce87b7ede5ecc433104c796beb102f9a54a6c91b" Jul 25 02:06:57.702851 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:06:57.702834 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5482912963bdf549debb17c8ce87b7ede5ecc433104c796beb102f9a54a6c91b\": container with ID starting with 5482912963bdf549debb17c8ce87b7ede5ecc433104c796beb102f9a54a6c91b not found: ID does not exist" containerID="5482912963bdf549debb17c8ce87b7ede5ecc433104c796beb102f9a54a6c91b" Jul 25 02:06:57.702910 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.702858 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5482912963bdf549debb17c8ce87b7ede5ecc433104c796beb102f9a54a6c91b"} err="failed to get container status \"5482912963bdf549debb17c8ce87b7ede5ecc433104c796beb102f9a54a6c91b\": rpc error: code = NotFound desc = could not find container \"5482912963bdf549debb17c8ce87b7ede5ecc433104c796beb102f9a54a6c91b\": container with ID starting with 5482912963bdf549debb17c8ce87b7ede5ecc433104c796beb102f9a54a6c91b not found: ID does not exist" Jul 25 02:06:57.708330 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.708306 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk"] Jul 25 02:06:57.711464 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.711441 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9b5fd-7754595d88-prqxk"] Jul 25 02:06:57.720223 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.720199 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8"] Jul 25 02:06:57.723666 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:57.723645 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9b5fd-5d98cdcf58-8gvr8"] Jul 25 02:06:59.382431 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:59.382403 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a74be98-bb0e-49fe-a5ac-24e876e4ec5e" path="/var/lib/kubelet/pods/1a74be98-bb0e-49fe-a5ac-24e876e4ec5e/volumes" Jul 25 02:06:59.382771 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:59.382739 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac1012ca-c394-4f5c-979a-0a84e99fe316" path="/var/lib/kubelet/pods/ac1012ca-c394-4f5c-979a-0a84e99fe316/volumes" Jul 25 02:06:59.383084 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:06:59.383070 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae7cdbd1-7fb7-4bec-915d-8744fc00bc28" path="/var/lib/kubelet/pods/ae7cdbd1-7fb7-4bec-915d-8744fc00bc28/volumes" Jul 25 02:07:23.916318 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:23.916284 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6"] Jul 25 02:07:23.917038 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:23.916985 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" podUID="4b8712ad-abee-4f21-b824-b9936c2b8c82" containerName="maas-api" containerID="cri-o://af8c08c0b185498432038427b44b29464f0a53a60cc7f3313a3e3617f0b08eed" gracePeriod=30 Jul 25 02:07:24.169925 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:24.169843 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" Jul 25 02:07:24.341483 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:24.341455 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8712ad-abee-4f21-b824-b9936c2b8c82-maas-api-tls\") pod \"4b8712ad-abee-4f21-b824-b9936c2b8c82\" (UID: \"4b8712ad-abee-4f21-b824-b9936c2b8c82\") " Jul 25 02:07:24.341666 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:24.341520 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbskn\" (UniqueName: \"kubernetes.io/projected/4b8712ad-abee-4f21-b824-b9936c2b8c82-kube-api-access-sbskn\") pod \"4b8712ad-abee-4f21-b824-b9936c2b8c82\" (UID: \"4b8712ad-abee-4f21-b824-b9936c2b8c82\") " Jul 25 02:07:24.341666 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:24.341550 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8712ad-abee-4f21-b824-b9936c2b8c82-openshift-service-ca\") pod \"4b8712ad-abee-4f21-b824-b9936c2b8c82\" (UID: \"4b8712ad-abee-4f21-b824-b9936c2b8c82\") " Jul 25 02:07:24.341972 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:24.341947 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8712ad-abee-4f21-b824-b9936c2b8c82-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4b8712ad-abee-4f21-b824-b9936c2b8c82" (UID: "4b8712ad-abee-4f21-b824-b9936c2b8c82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:07:24.343639 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:24.343615 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b8712ad-abee-4f21-b824-b9936c2b8c82-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4b8712ad-abee-4f21-b824-b9936c2b8c82" (UID: "4b8712ad-abee-4f21-b824-b9936c2b8c82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 02:07:24.343721 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:24.343618 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8712ad-abee-4f21-b824-b9936c2b8c82-kube-api-access-sbskn" (OuterVolumeSpecName: "kube-api-access-sbskn") pod "4b8712ad-abee-4f21-b824-b9936c2b8c82" (UID: "4b8712ad-abee-4f21-b824-b9936c2b8c82"). InnerVolumeSpecName "kube-api-access-sbskn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:07:24.442089 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:24.442035 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8712ad-abee-4f21-b824-b9936c2b8c82-openshift-service-ca\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:07:24.442089 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:24.442056 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8712ad-abee-4f21-b824-b9936c2b8c82-maas-api-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:07:24.442089 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:24.442065 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbskn\" (UniqueName: \"kubernetes.io/projected/4b8712ad-abee-4f21-b824-b9936c2b8c82-kube-api-access-sbskn\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:07:24.819964 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:24.819877 2554 generic.go:356] "Generic (PLEG): container finished" podID="4b8712ad-abee-4f21-b824-b9936c2b8c82" containerID="af8c08c0b185498432038427b44b29464f0a53a60cc7f3313a3e3617f0b08eed" exitCode=0 Jul 25 02:07:24.819964 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:24.819917 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" event={"ID":"4b8712ad-abee-4f21-b824-b9936c2b8c82","Type":"ContainerDied","Data":"af8c08c0b185498432038427b44b29464f0a53a60cc7f3313a3e3617f0b08eed"} Jul 25 02:07:24.819964 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:24.819922 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" Jul 25 02:07:24.819964 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:24.819953 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6" event={"ID":"4b8712ad-abee-4f21-b824-b9936c2b8c82","Type":"ContainerDied","Data":"2c9eb856391b5cbff4cd0b01db7555c16dc03269d024ba79c829655e8a8fb2dc"} Jul 25 02:07:24.820276 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:24.819973 2554 scope.go:117] "RemoveContainer" containerID="af8c08c0b185498432038427b44b29464f0a53a60cc7f3313a3e3617f0b08eed" Jul 25 02:07:24.829723 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:24.829706 2554 scope.go:117] "RemoveContainer" containerID="af8c08c0b185498432038427b44b29464f0a53a60cc7f3313a3e3617f0b08eed" Jul 25 02:07:24.830019 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:07:24.829995 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af8c08c0b185498432038427b44b29464f0a53a60cc7f3313a3e3617f0b08eed\": container with ID starting with af8c08c0b185498432038427b44b29464f0a53a60cc7f3313a3e3617f0b08eed not found: ID does not exist" containerID="af8c08c0b185498432038427b44b29464f0a53a60cc7f3313a3e3617f0b08eed" Jul 25 02:07:24.830088 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:24.830030 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af8c08c0b185498432038427b44b29464f0a53a60cc7f3313a3e3617f0b08eed"} err="failed to get container status \"af8c08c0b185498432038427b44b29464f0a53a60cc7f3313a3e3617f0b08eed\": rpc error: code = NotFound desc = could not find container \"af8c08c0b185498432038427b44b29464f0a53a60cc7f3313a3e3617f0b08eed\": container with ID starting with af8c08c0b185498432038427b44b29464f0a53a60cc7f3313a3e3617f0b08eed not found: ID does not exist" Jul 25 02:07:24.839464 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:24.839445 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6"] Jul 25 02:07:24.842730 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:24.842710 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6fd1ef-bbd49b679-8kjb6"] Jul 25 02:07:25.381858 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:25.381823 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b8712ad-abee-4f21-b824-b9936c2b8c82" path="/var/lib/kubelet/pods/4b8712ad-abee-4f21-b824-b9936c2b8c82/volumes" Jul 25 02:07:38.924305 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:38.924272 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq"] Jul 25 02:07:38.924707 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:38.924621 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a74be98-bb0e-49fe-a5ac-24e876e4ec5e" containerName="maas-api" Jul 25 02:07:38.924707 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:38.924631 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a74be98-bb0e-49fe-a5ac-24e876e4ec5e" containerName="maas-api" Jul 25 02:07:38.924707 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:38.924640 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac1012ca-c394-4f5c-979a-0a84e99fe316" containerName="payload-pre-processing" Jul 25 02:07:38.924707 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:38.924646 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac1012ca-c394-4f5c-979a-0a84e99fe316" containerName="payload-pre-processing" Jul 25 02:07:38.924707 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:38.924658 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae7cdbd1-7fb7-4bec-915d-8744fc00bc28" containerName="payload-processing" Jul 25 02:07:38.924707 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:38.924663 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae7cdbd1-7fb7-4bec-915d-8744fc00bc28" containerName="payload-processing" Jul 25 02:07:38.924707 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:38.924672 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b8712ad-abee-4f21-b824-b9936c2b8c82" containerName="maas-api" Jul 25 02:07:38.924707 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:38.924676 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b8712ad-abee-4f21-b824-b9936c2b8c82" containerName="maas-api" Jul 25 02:07:38.924993 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:38.924730 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b8712ad-abee-4f21-b824-b9936c2b8c82" containerName="maas-api" Jul 25 02:07:38.924993 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:38.924737 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a74be98-bb0e-49fe-a5ac-24e876e4ec5e" containerName="maas-api" Jul 25 02:07:38.924993 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:38.924745 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae7cdbd1-7fb7-4bec-915d-8744fc00bc28" containerName="payload-processing" Jul 25 02:07:38.924993 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:38.924751 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac1012ca-c394-4f5c-979a-0a84e99fe316" containerName="payload-pre-processing" Jul 25 02:07:38.929369 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:38.929348 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" Jul 25 02:07:38.931502 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:38.931470 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-7e06ec-serving-cert\"" Jul 25 02:07:38.938885 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:38.938823 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq"] Jul 25 02:07:38.970579 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:38.970551 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6f1989-725f-4030-889b-4223aa4e084e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq\" (UID: \"6b6f1989-725f-4030-889b-4223aa4e084e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" Jul 25 02:07:38.970717 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:38.970596 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w4g5\" (UniqueName: \"kubernetes.io/projected/6b6f1989-725f-4030-889b-4223aa4e084e-kube-api-access-2w4g5\") pod \"maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq\" (UID: \"6b6f1989-725f-4030-889b-4223aa4e084e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" Jul 25 02:07:38.970717 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:38.970695 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6f1989-725f-4030-889b-4223aa4e084e-maas-api-tls\") pod \"maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq\" (UID: \"6b6f1989-725f-4030-889b-4223aa4e084e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" Jul 25 02:07:39.071642 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.071589 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w4g5\" (UniqueName: \"kubernetes.io/projected/6b6f1989-725f-4030-889b-4223aa4e084e-kube-api-access-2w4g5\") pod \"maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq\" (UID: \"6b6f1989-725f-4030-889b-4223aa4e084e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" Jul 25 02:07:39.071847 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.071678 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6f1989-725f-4030-889b-4223aa4e084e-maas-api-tls\") pod \"maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq\" (UID: \"6b6f1989-725f-4030-889b-4223aa4e084e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" Jul 25 02:07:39.071847 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.071760 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6f1989-725f-4030-889b-4223aa4e084e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq\" (UID: \"6b6f1989-725f-4030-889b-4223aa4e084e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" Jul 25 02:07:39.071973 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:07:39.071852 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-serving-cert: secret "maas-api-e2e-shared-a-7e06ec-serving-cert" not found Jul 25 02:07:39.071973 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:07:39.071945 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6f1989-725f-4030-889b-4223aa4e084e-maas-api-tls podName:6b6f1989-725f-4030-889b-4223aa4e084e nodeName:}" failed. No retries permitted until 2026-07-25 02:07:39.571922788 +0000 UTC m=+3400.693408620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6b6f1989-725f-4030-889b-4223aa4e084e-maas-api-tls") pod "maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" (UID: "6b6f1989-725f-4030-889b-4223aa4e084e") : secret "maas-api-e2e-shared-a-7e06ec-serving-cert" not found Jul 25 02:07:39.072506 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.072487 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6f1989-725f-4030-889b-4223aa4e084e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq\" (UID: \"6b6f1989-725f-4030-889b-4223aa4e084e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" Jul 25 02:07:39.080168 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.080144 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w4g5\" (UniqueName: \"kubernetes.io/projected/6b6f1989-725f-4030-889b-4223aa4e084e-kube-api-access-2w4g5\") pod \"maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq\" (UID: \"6b6f1989-725f-4030-889b-4223aa4e084e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" Jul 25 02:07:39.539437 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.539403 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d"] Jul 25 02:07:39.542950 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.542934 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d" Jul 25 02:07:39.545216 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.545194 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-7e06ec\"" Jul 25 02:07:39.545340 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.545200 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-7e06ec-dockercfg-484wn\"" Jul 25 02:07:39.551384 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.551364 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d"] Jul 25 02:07:39.577292 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.577262 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6f1989-725f-4030-889b-4223aa4e084e-maas-api-tls\") pod \"maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq\" (UID: \"6b6f1989-725f-4030-889b-4223aa4e084e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" Jul 25 02:07:39.577422 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.577332 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65992df1-2c85-4b51-a9ca-204b65045a38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d\" (UID: \"65992df1-2c85-4b51-a9ca-204b65045a38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d" Jul 25 02:07:39.577422 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.577392 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwql2\" (UniqueName: \"kubernetes.io/projected/65992df1-2c85-4b51-a9ca-204b65045a38-kube-api-access-fwql2\") pod \"payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d\" (UID: \"65992df1-2c85-4b51-a9ca-204b65045a38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d" Jul 25 02:07:39.580144 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.580114 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6f1989-725f-4030-889b-4223aa4e084e-maas-api-tls\") pod \"maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq\" (UID: \"6b6f1989-725f-4030-889b-4223aa4e084e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" Jul 25 02:07:39.584545 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.581764 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6"] Jul 25 02:07:39.587452 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.586874 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6" Jul 25 02:07:39.593578 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.593553 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6"] Jul 25 02:07:39.678329 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.678298 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwql2\" (UniqueName: \"kubernetes.io/projected/65992df1-2c85-4b51-a9ca-204b65045a38-kube-api-access-fwql2\") pod \"payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d\" (UID: \"65992df1-2c85-4b51-a9ca-204b65045a38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d" Jul 25 02:07:39.678487 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.678360 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9111eb02-361f-43b0-9bf1-deaf189a12b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6\" (UID: \"9111eb02-361f-43b0-9bf1-deaf189a12b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6" Jul 25 02:07:39.678487 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.678425 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65992df1-2c85-4b51-a9ca-204b65045a38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d\" (UID: \"65992df1-2c85-4b51-a9ca-204b65045a38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d" Jul 25 02:07:39.678579 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.678492 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7r5v\" (UniqueName: \"kubernetes.io/projected/9111eb02-361f-43b0-9bf1-deaf189a12b7-kube-api-access-p7r5v\") pod \"payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6\" (UID: \"9111eb02-361f-43b0-9bf1-deaf189a12b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6" Jul 25 02:07:39.679140 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.679122 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65992df1-2c85-4b51-a9ca-204b65045a38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d\" (UID: \"65992df1-2c85-4b51-a9ca-204b65045a38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d" Jul 25 02:07:39.685768 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.685750 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwql2\" (UniqueName: \"kubernetes.io/projected/65992df1-2c85-4b51-a9ca-204b65045a38-kube-api-access-fwql2\") pod \"payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d\" (UID: \"65992df1-2c85-4b51-a9ca-204b65045a38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d" Jul 25 02:07:39.779237 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.779209 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9111eb02-361f-43b0-9bf1-deaf189a12b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6\" (UID: \"9111eb02-361f-43b0-9bf1-deaf189a12b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6" Jul 25 02:07:39.779454 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.779304 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7r5v\" (UniqueName: \"kubernetes.io/projected/9111eb02-361f-43b0-9bf1-deaf189a12b7-kube-api-access-p7r5v\") pod \"payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6\" (UID: \"9111eb02-361f-43b0-9bf1-deaf189a12b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6" Jul 25 02:07:39.779925 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.779906 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9111eb02-361f-43b0-9bf1-deaf189a12b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6\" (UID: \"9111eb02-361f-43b0-9bf1-deaf189a12b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6" Jul 25 02:07:39.787133 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.787102 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7r5v\" (UniqueName: \"kubernetes.io/projected/9111eb02-361f-43b0-9bf1-deaf189a12b7-kube-api-access-p7r5v\") pod \"payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6\" (UID: \"9111eb02-361f-43b0-9bf1-deaf189a12b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6" Jul 25 02:07:39.842171 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.842094 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" Jul 25 02:07:39.855305 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.855280 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d" Jul 25 02:07:39.901340 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:39.901081 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6" Jul 25 02:07:40.008978 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:40.008902 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq"] Jul 25 02:07:40.010656 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:07:40.010617 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6f1989_725f_4030_889b_4223aa4e084e.slice/crio-0908303fa056da02b7b87f2bbfcc8936b1b0b5b24b8d7fdf87d457d6f83e47e9 WatchSource:0}: Error finding container 0908303fa056da02b7b87f2bbfcc8936b1b0b5b24b8d7fdf87d457d6f83e47e9: Status 404 returned error can't find the container with id 0908303fa056da02b7b87f2bbfcc8936b1b0b5b24b8d7fdf87d457d6f83e47e9 Jul 25 02:07:40.035424 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:40.035396 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d"] Jul 25 02:07:40.039692 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:07:40.039661 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65992df1_2c85_4b51_a9ca_204b65045a38.slice/crio-80cac9fd8c2dc6c3121eb14d46815916721299f710c211d566b9be2e0dfea2b2 WatchSource:0}: Error finding container 80cac9fd8c2dc6c3121eb14d46815916721299f710c211d566b9be2e0dfea2b2: Status 404 returned error can't find the container with id 80cac9fd8c2dc6c3121eb14d46815916721299f710c211d566b9be2e0dfea2b2 Jul 25 02:07:40.273328 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:40.273290 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6"] Jul 25 02:07:40.276342 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:07:40.276304 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9111eb02_361f_43b0_9bf1_deaf189a12b7.slice/crio-d5010021d3aa2ec16955c769f76fb36da7128e7f1a2b7b2cbfce66c1abd87dba WatchSource:0}: Error finding container d5010021d3aa2ec16955c769f76fb36da7128e7f1a2b7b2cbfce66c1abd87dba: Status 404 returned error can't find the container with id d5010021d3aa2ec16955c769f76fb36da7128e7f1a2b7b2cbfce66c1abd87dba Jul 25 02:07:40.914989 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:40.914959 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" event={"ID":"6b6f1989-725f-4030-889b-4223aa4e084e","Type":"ContainerStarted","Data":"114d3e1ed01eb8f9fedc495bdf7fe2b15038f3be39ebd54ad0a8893948a7c895"} Jul 25 02:07:40.915177 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:40.915009 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" event={"ID":"6b6f1989-725f-4030-889b-4223aa4e084e","Type":"ContainerStarted","Data":"0908303fa056da02b7b87f2bbfcc8936b1b0b5b24b8d7fdf87d457d6f83e47e9"} Jul 25 02:07:40.915177 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:40.915167 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" Jul 25 02:07:40.916720 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:40.916696 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d" event={"ID":"65992df1-2c85-4b51-a9ca-204b65045a38","Type":"ContainerStarted","Data":"52f96081cfca5c02c9b8033c814b64a5f2d0469f5b1840c813b6e565f46203f7"} Jul 25 02:07:40.916865 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:40.916733 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d" event={"ID":"65992df1-2c85-4b51-a9ca-204b65045a38","Type":"ContainerStarted","Data":"80cac9fd8c2dc6c3121eb14d46815916721299f710c211d566b9be2e0dfea2b2"} Jul 25 02:07:40.916865 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:40.916794 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d" Jul 25 02:07:40.918299 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:40.918277 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6" event={"ID":"9111eb02-361f-43b0-9bf1-deaf189a12b7","Type":"ContainerStarted","Data":"f163f2b17731edb4aa1607eeb58d7e0874b362ff599d645fda8cde1c9a84731e"} Jul 25 02:07:40.918421 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:40.918312 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6" event={"ID":"9111eb02-361f-43b0-9bf1-deaf189a12b7","Type":"ContainerStarted","Data":"d5010021d3aa2ec16955c769f76fb36da7128e7f1a2b7b2cbfce66c1abd87dba"} Jul 25 02:07:40.918421 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:40.918342 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6" Jul 25 02:07:40.934208 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:40.934154 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" podStartSLOduration=2.433384233 podStartE2EDuration="2.934142208s" totalImagesPullingTime="500.757975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:07:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:07:40.931563733 +0000 UTC m=+3402.053049583" watchObservedRunningTime="2026-07-25 02:07:40.934142208 +0000 UTC m=+3402.055628057" Jul 25 02:07:40.946918 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:40.946876 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d" podStartSLOduration=1.946863709 podStartE2EDuration="1.946863709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:07:40.946365786 +0000 UTC m=+3402.067851635" watchObservedRunningTime="2026-07-25 02:07:40.946863709 +0000 UTC m=+3402.068349559" Jul 25 02:07:40.961028 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:40.960987 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6" podStartSLOduration=1.9609751050000002 podStartE2EDuration="1.960975105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:07:40.959759495 +0000 UTC m=+3402.081245345" watchObservedRunningTime="2026-07-25 02:07:40.960975105 +0000 UTC m=+3402.082460954" Jul 25 02:07:46.929845 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:46.929789 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" Jul 25 02:07:47.030270 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:47.030234 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29"] Jul 25 02:07:47.034560 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:47.034539 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" Jul 25 02:07:47.036780 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:47.036755 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-575aa7-serving-cert\"" Jul 25 02:07:47.043197 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:47.042688 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29"] Jul 25 02:07:47.148295 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:47.148258 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q626s\" (UniqueName: \"kubernetes.io/projected/455cb900-e4d1-4e61-b891-7d3011757d19-kube-api-access-q626s\") pod \"maas-api-e2e-shared-b-575aa7-6589cfd776-87z29\" (UID: \"455cb900-e4d1-4e61-b891-7d3011757d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" Jul 25 02:07:47.148465 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:47.148335 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/455cb900-e4d1-4e61-b891-7d3011757d19-maas-api-tls\") pod \"maas-api-e2e-shared-b-575aa7-6589cfd776-87z29\" (UID: \"455cb900-e4d1-4e61-b891-7d3011757d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" Jul 25 02:07:47.148465 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:47.148371 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/455cb900-e4d1-4e61-b891-7d3011757d19-openshift-service-ca\") pod \"maas-api-e2e-shared-b-575aa7-6589cfd776-87z29\" (UID: \"455cb900-e4d1-4e61-b891-7d3011757d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" Jul 25 02:07:47.249938 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:47.249857 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q626s\" (UniqueName: \"kubernetes.io/projected/455cb900-e4d1-4e61-b891-7d3011757d19-kube-api-access-q626s\") pod \"maas-api-e2e-shared-b-575aa7-6589cfd776-87z29\" (UID: \"455cb900-e4d1-4e61-b891-7d3011757d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" Jul 25 02:07:47.249938 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:47.249934 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/455cb900-e4d1-4e61-b891-7d3011757d19-maas-api-tls\") pod \"maas-api-e2e-shared-b-575aa7-6589cfd776-87z29\" (UID: \"455cb900-e4d1-4e61-b891-7d3011757d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" Jul 25 02:07:47.250132 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:47.249954 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/455cb900-e4d1-4e61-b891-7d3011757d19-openshift-service-ca\") pod \"maas-api-e2e-shared-b-575aa7-6589cfd776-87z29\" (UID: \"455cb900-e4d1-4e61-b891-7d3011757d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" Jul 25 02:07:47.250659 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:47.250632 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/455cb900-e4d1-4e61-b891-7d3011757d19-openshift-service-ca\") pod \"maas-api-e2e-shared-b-575aa7-6589cfd776-87z29\" (UID: \"455cb900-e4d1-4e61-b891-7d3011757d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" Jul 25 02:07:47.252493 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:47.252465 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/455cb900-e4d1-4e61-b891-7d3011757d19-maas-api-tls\") pod \"maas-api-e2e-shared-b-575aa7-6589cfd776-87z29\" (UID: \"455cb900-e4d1-4e61-b891-7d3011757d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" Jul 25 02:07:47.257858 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:47.257834 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q626s\" (UniqueName: \"kubernetes.io/projected/455cb900-e4d1-4e61-b891-7d3011757d19-kube-api-access-q626s\") pod \"maas-api-e2e-shared-b-575aa7-6589cfd776-87z29\" (UID: \"455cb900-e4d1-4e61-b891-7d3011757d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" Jul 25 02:07:47.348845 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:47.348790 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" Jul 25 02:07:47.520496 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:47.520417 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29"] Jul 25 02:07:47.521729 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:07:47.521709 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455cb900_e4d1_4e61_b891_7d3011757d19.slice/crio-17267ede790de5141ccaae961444f5a80d2a362a291453503cf05ff1064cbcc8 WatchSource:0}: Error finding container 17267ede790de5141ccaae961444f5a80d2a362a291453503cf05ff1064cbcc8: Status 404 returned error can't find the container with id 17267ede790de5141ccaae961444f5a80d2a362a291453503cf05ff1064cbcc8 Jul 25 02:07:47.958041 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:47.957996 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" event={"ID":"455cb900-e4d1-4e61-b891-7d3011757d19","Type":"ContainerStarted","Data":"479150b9455207153dd411e8c5740a7cd953477d1cb5b63e5cfa494d10dedbba"} Jul 25 02:07:47.958414 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:47.958051 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" event={"ID":"455cb900-e4d1-4e61-b891-7d3011757d19","Type":"ContainerStarted","Data":"17267ede790de5141ccaae961444f5a80d2a362a291453503cf05ff1064cbcc8"} Jul 25 02:07:47.958414 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:47.958144 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" Jul 25 02:07:47.978368 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:47.978321 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" podStartSLOduration=0.664134376 podStartE2EDuration="978.306509ms" totalImagesPullingTime="314.172133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:07:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:07:47.975398622 +0000 UTC m=+3409.096884474" watchObservedRunningTime="2026-07-25 02:07:47.978306509 +0000 UTC m=+3409.099792359" Jul 25 02:07:51.926912 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:51.926834 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6" Jul 25 02:07:51.927295 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:51.926915 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d" Jul 25 02:07:53.969849 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:07:53.969819 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" Jul 25 02:16:45.046002 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.045968 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw"] Jul 25 02:16:45.049458 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.049441 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" Jul 25 02:16:45.051891 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.051865 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-67eb46-serving-cert\"" Jul 25 02:16:45.061274 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.061252 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw"] Jul 25 02:16:45.128552 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.128519 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b035564b-572c-4d98-89d5-933bc4ee9749-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw\" (UID: \"b035564b-572c-4d98-89d5-933bc4ee9749\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" Jul 25 02:16:45.128728 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.128570 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw9nw\" (UniqueName: \"kubernetes.io/projected/b035564b-572c-4d98-89d5-933bc4ee9749-kube-api-access-fw9nw\") pod \"maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw\" (UID: \"b035564b-572c-4d98-89d5-933bc4ee9749\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" Jul 25 02:16:45.128728 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.128642 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b035564b-572c-4d98-89d5-933bc4ee9749-maas-api-tls\") pod \"maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw\" (UID: \"b035564b-572c-4d98-89d5-933bc4ee9749\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" Jul 25 02:16:45.230040 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.230012 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b035564b-572c-4d98-89d5-933bc4ee9749-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw\" (UID: \"b035564b-572c-4d98-89d5-933bc4ee9749\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" Jul 25 02:16:45.230040 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.230051 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw9nw\" (UniqueName: \"kubernetes.io/projected/b035564b-572c-4d98-89d5-933bc4ee9749-kube-api-access-fw9nw\") pod \"maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw\" (UID: \"b035564b-572c-4d98-89d5-933bc4ee9749\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" Jul 25 02:16:45.230268 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.230085 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b035564b-572c-4d98-89d5-933bc4ee9749-maas-api-tls\") pod \"maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw\" (UID: \"b035564b-572c-4d98-89d5-933bc4ee9749\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" Jul 25 02:16:45.230764 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.230703 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b035564b-572c-4d98-89d5-933bc4ee9749-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw\" (UID: \"b035564b-572c-4d98-89d5-933bc4ee9749\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" Jul 25 02:16:45.232621 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.232600 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b035564b-572c-4d98-89d5-933bc4ee9749-maas-api-tls\") pod \"maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw\" (UID: \"b035564b-572c-4d98-89d5-933bc4ee9749\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" Jul 25 02:16:45.238598 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.238575 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw9nw\" (UniqueName: \"kubernetes.io/projected/b035564b-572c-4d98-89d5-933bc4ee9749-kube-api-access-fw9nw\") pod \"maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw\" (UID: \"b035564b-572c-4d98-89d5-933bc4ee9749\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" Jul 25 02:16:45.363676 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.363602 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" Jul 25 02:16:45.505886 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.505863 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw"] Jul 25 02:16:45.508420 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:16:45.508393 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb035564b_572c_4d98_89d5_933bc4ee9749.slice/crio-39a4fbc4cedc4ecd80b1a861809f8810ebf23f84c95ffea4ef3616cc5c1452f3 WatchSource:0}: Error finding container 39a4fbc4cedc4ecd80b1a861809f8810ebf23f84c95ffea4ef3616cc5c1452f3: Status 404 returned error can't find the container with id 39a4fbc4cedc4ecd80b1a861809f8810ebf23f84c95ffea4ef3616cc5c1452f3 Jul 25 02:16:45.509813 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.509781 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 25 02:16:45.726041 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.726006 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b"] Jul 25 02:16:45.730381 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.730364 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b" Jul 25 02:16:45.732377 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.732354 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-67eb46\"" Jul 25 02:16:45.732483 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.732461 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-67eb46-dockercfg-z4wlg\"" Jul 25 02:16:45.739659 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.739636 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b"] Jul 25 02:16:45.823739 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.823706 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" event={"ID":"b035564b-572c-4d98-89d5-933bc4ee9749","Type":"ContainerStarted","Data":"39a4fbc4cedc4ecd80b1a861809f8810ebf23f84c95ffea4ef3616cc5c1452f3"} Jul 25 02:16:45.835155 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.835131 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859cced6-5143-4f87-8243-10c94bdc9a64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b\" (UID: \"859cced6-5143-4f87-8243-10c94bdc9a64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b" Jul 25 02:16:45.835241 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.835182 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgbz6\" (UniqueName: \"kubernetes.io/projected/859cced6-5143-4f87-8243-10c94bdc9a64-kube-api-access-tgbz6\") pod \"payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b\" (UID: \"859cced6-5143-4f87-8243-10c94bdc9a64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b" Jul 25 02:16:45.936395 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.936375 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859cced6-5143-4f87-8243-10c94bdc9a64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b\" (UID: \"859cced6-5143-4f87-8243-10c94bdc9a64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b" Jul 25 02:16:45.936478 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.936424 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgbz6\" (UniqueName: \"kubernetes.io/projected/859cced6-5143-4f87-8243-10c94bdc9a64-kube-api-access-tgbz6\") pod \"payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b\" (UID: \"859cced6-5143-4f87-8243-10c94bdc9a64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b" Jul 25 02:16:45.937090 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.937072 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859cced6-5143-4f87-8243-10c94bdc9a64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b\" (UID: \"859cced6-5143-4f87-8243-10c94bdc9a64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b" Jul 25 02:16:45.946321 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:45.944019 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgbz6\" (UniqueName: \"kubernetes.io/projected/859cced6-5143-4f87-8243-10c94bdc9a64-kube-api-access-tgbz6\") pod \"payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b\" (UID: \"859cced6-5143-4f87-8243-10c94bdc9a64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b" Jul 25 02:16:46.042339 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:46.042310 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b" Jul 25 02:16:46.180617 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:46.180593 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b"] Jul 25 02:16:46.182563 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:16:46.182531 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859cced6_5143_4f87_8243_10c94bdc9a64.slice/crio-8962a92c814dd4b06e68e4c4a77833b303a044f6f86681e6a521880ac6ebac31 WatchSource:0}: Error finding container 8962a92c814dd4b06e68e4c4a77833b303a044f6f86681e6a521880ac6ebac31: Status 404 returned error can't find the container with id 8962a92c814dd4b06e68e4c4a77833b303a044f6f86681e6a521880ac6ebac31 Jul 25 02:16:46.830947 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:46.830902 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b" event={"ID":"859cced6-5143-4f87-8243-10c94bdc9a64","Type":"ContainerStarted","Data":"c91bbe131fbf210262a96eb103f0a64f678eb4725a3e38ae3b4995cb5baab330"} Jul 25 02:16:46.831133 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:46.830975 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b" Jul 25 02:16:46.831133 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:46.830996 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b" event={"ID":"859cced6-5143-4f87-8243-10c94bdc9a64","Type":"ContainerStarted","Data":"8962a92c814dd4b06e68e4c4a77833b303a044f6f86681e6a521880ac6ebac31"} Jul 25 02:16:46.832497 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:46.832474 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" event={"ID":"b035564b-572c-4d98-89d5-933bc4ee9749","Type":"ContainerStarted","Data":"71872855c3fbc3cab5b55c7ead59f35a261e35f1397044b87c337ed1d4168d97"} Jul 25 02:16:46.832612 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:46.832582 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" Jul 25 02:16:46.853873 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:46.853817 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b" podStartSLOduration=1.853784306 podStartE2EDuration="1.853784306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:16:46.851729308 +0000 UTC m=+3947.973215152" watchObservedRunningTime="2026-07-25 02:16:46.853784306 +0000 UTC m=+3947.975270157" Jul 25 02:16:46.867879 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:46.867834 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" podStartSLOduration=1.4488547760000001 podStartE2EDuration="1.86782005s" totalImagesPullingTime="418.965274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:16:46.865992974 +0000 UTC m=+3947.987478825" watchObservedRunningTime="2026-07-25 02:16:46.86782005 +0000 UTC m=+3947.989305900" Jul 25 02:16:52.844008 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:52.843929 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" Jul 25 02:16:57.839786 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:16:57.839755 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b" Jul 25 02:17:07.632985 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:07.632950 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5"] Jul 25 02:17:07.637920 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:07.637875 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" Jul 25 02:17:07.640542 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:07.640511 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-dc8abe-serving-cert\"" Jul 25 02:17:07.648016 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:07.647491 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5"] Jul 25 02:17:07.826767 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:07.826732 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91e835f5-a77c-4fa7-ac4f-0a91d4583f70-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5\" (UID: \"91e835f5-a77c-4fa7-ac4f-0a91d4583f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" Jul 25 02:17:07.826950 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:07.826863 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e835f5-a77c-4fa7-ac4f-0a91d4583f70-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5\" (UID: \"91e835f5-a77c-4fa7-ac4f-0a91d4583f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" Jul 25 02:17:07.826950 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:07.826911 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72hjz\" (UniqueName: \"kubernetes.io/projected/91e835f5-a77c-4fa7-ac4f-0a91d4583f70-kube-api-access-72hjz\") pod \"maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5\" (UID: \"91e835f5-a77c-4fa7-ac4f-0a91d4583f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" Jul 25 02:17:07.928241 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:07.928214 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91e835f5-a77c-4fa7-ac4f-0a91d4583f70-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5\" (UID: \"91e835f5-a77c-4fa7-ac4f-0a91d4583f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" Jul 25 02:17:07.928429 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:07.928267 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e835f5-a77c-4fa7-ac4f-0a91d4583f70-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5\" (UID: \"91e835f5-a77c-4fa7-ac4f-0a91d4583f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" Jul 25 02:17:07.928429 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:07.928284 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72hjz\" (UniqueName: \"kubernetes.io/projected/91e835f5-a77c-4fa7-ac4f-0a91d4583f70-kube-api-access-72hjz\") pod \"maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5\" (UID: \"91e835f5-a77c-4fa7-ac4f-0a91d4583f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" Jul 25 02:17:07.928860 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:07.928837 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e835f5-a77c-4fa7-ac4f-0a91d4583f70-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5\" (UID: \"91e835f5-a77c-4fa7-ac4f-0a91d4583f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" Jul 25 02:17:07.930927 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:07.930904 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91e835f5-a77c-4fa7-ac4f-0a91d4583f70-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5\" (UID: \"91e835f5-a77c-4fa7-ac4f-0a91d4583f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" Jul 25 02:17:07.935939 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:07.935919 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72hjz\" (UniqueName: \"kubernetes.io/projected/91e835f5-a77c-4fa7-ac4f-0a91d4583f70-kube-api-access-72hjz\") pod \"maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5\" (UID: \"91e835f5-a77c-4fa7-ac4f-0a91d4583f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" Jul 25 02:17:07.952905 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:07.952885 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" Jul 25 02:17:08.096971 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:08.096945 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5"] Jul 25 02:17:08.098784 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:17:08.098743 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e835f5_a77c_4fa7_ac4f_0a91d4583f70.slice/crio-1e07ca8deaa11b63d2426ff151f0097c45e48abd58b7fcea64c6fac051c427b9 WatchSource:0}: Error finding container 1e07ca8deaa11b63d2426ff151f0097c45e48abd58b7fcea64c6fac051c427b9: Status 404 returned error can't find the container with id 1e07ca8deaa11b63d2426ff151f0097c45e48abd58b7fcea64c6fac051c427b9 Jul 25 02:17:08.965110 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:08.965076 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" event={"ID":"91e835f5-a77c-4fa7-ac4f-0a91d4583f70","Type":"ContainerStarted","Data":"0dfb6f560a2d5c92edc93b2a0c74f78b99ed69d3703e6e31268119aae65e5519"} Jul 25 02:17:08.965520 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:08.965126 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" event={"ID":"91e835f5-a77c-4fa7-ac4f-0a91d4583f70","Type":"ContainerStarted","Data":"1e07ca8deaa11b63d2426ff151f0097c45e48abd58b7fcea64c6fac051c427b9"} Jul 25 02:17:08.965520 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:08.965207 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" Jul 25 02:17:08.985142 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:08.985097 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" podStartSLOduration=1.630062391 podStartE2EDuration="1.985081336s" totalImagesPullingTime="355.018945ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:17:08.984109619 +0000 UTC m=+3970.105595468" watchObservedRunningTime="2026-07-25 02:17:08.985081336 +0000 UTC m=+3970.106567238" Jul 25 02:17:14.976049 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:14.976017 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" Jul 25 02:17:28.057620 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:17:28.057585 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-cpblz_834d00ec-f37a-4e16-a9c1-c7c77d337fcf/payload-processing/0.log" Jul 25 02:18:28.321000 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:28.320917 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b_859cced6-5143-4f87-8243-10c94bdc9a64/payload-processing/0.log" Jul 25 02:18:28.441072 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:28.441035 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-cpblz_834d00ec-f37a-4e16-a9c1-c7c77d337fcf/payload-processing/0.log" Jul 25 02:18:33.347744 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:33.347704 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg"] Jul 25 02:18:33.351471 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:33.351452 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" Jul 25 02:18:33.353532 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:33.353506 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-389518-serving-cert\"" Jul 25 02:18:33.359030 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:33.359005 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg"] Jul 25 02:18:33.476645 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:33.476609 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7h8b\" (UniqueName: \"kubernetes.io/projected/a6e00bef-b86c-4841-a735-fce8fa46b5aa-kube-api-access-s7h8b\") pod \"maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg\" (UID: \"a6e00bef-b86c-4841-a735-fce8fa46b5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" Jul 25 02:18:33.476846 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:33.476743 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e00bef-b86c-4841-a735-fce8fa46b5aa-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg\" (UID: \"a6e00bef-b86c-4841-a735-fce8fa46b5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" Jul 25 02:18:33.476846 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:33.476796 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e00bef-b86c-4841-a735-fce8fa46b5aa-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg\" (UID: \"a6e00bef-b86c-4841-a735-fce8fa46b5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" Jul 25 02:18:33.577375 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:33.577344 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e00bef-b86c-4841-a735-fce8fa46b5aa-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg\" (UID: \"a6e00bef-b86c-4841-a735-fce8fa46b5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" Jul 25 02:18:33.577539 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:33.577393 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e00bef-b86c-4841-a735-fce8fa46b5aa-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg\" (UID: \"a6e00bef-b86c-4841-a735-fce8fa46b5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" Jul 25 02:18:33.577539 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:18:33.577515 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-serving-cert: secret "maas-api-e2e-ipp-cleanup-389518-serving-cert" not found Jul 25 02:18:33.577625 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:33.577579 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7h8b\" (UniqueName: \"kubernetes.io/projected/a6e00bef-b86c-4841-a735-fce8fa46b5aa-kube-api-access-s7h8b\") pod \"maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg\" (UID: \"a6e00bef-b86c-4841-a735-fce8fa46b5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" Jul 25 02:18:33.577625 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:18:33.577601 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e00bef-b86c-4841-a735-fce8fa46b5aa-maas-api-tls podName:a6e00bef-b86c-4841-a735-fce8fa46b5aa nodeName:}" failed. No retries permitted until 2026-07-25 02:18:34.077579652 +0000 UTC m=+4055.199065484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a6e00bef-b86c-4841-a735-fce8fa46b5aa-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" (UID: "a6e00bef-b86c-4841-a735-fce8fa46b5aa") : secret "maas-api-e2e-ipp-cleanup-389518-serving-cert" not found Jul 25 02:18:33.578119 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:33.578097 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e00bef-b86c-4841-a735-fce8fa46b5aa-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg\" (UID: \"a6e00bef-b86c-4841-a735-fce8fa46b5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" Jul 25 02:18:33.586305 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:33.586279 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7h8b\" (UniqueName: \"kubernetes.io/projected/a6e00bef-b86c-4841-a735-fce8fa46b5aa-kube-api-access-s7h8b\") pod \"maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg\" (UID: \"a6e00bef-b86c-4841-a735-fce8fa46b5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" Jul 25 02:18:33.932097 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:33.932063 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg"] Jul 25 02:18:33.935850 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:33.935833 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg" Jul 25 02:18:33.937980 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:33.937947 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-389518\"" Jul 25 02:18:33.937980 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:33.937950 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-389518-dockercfg-w7l9s\"" Jul 25 02:18:33.945756 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:33.945733 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg"] Jul 25 02:18:34.081519 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:34.081483 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e00bef-b86c-4841-a735-fce8fa46b5aa-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg\" (UID: \"a6e00bef-b86c-4841-a735-fce8fa46b5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" Jul 25 02:18:34.081718 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:34.081532 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be105f51-e248-437b-b44d-0e6303ddcd92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg\" (UID: \"be105f51-e248-437b-b44d-0e6303ddcd92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg" Jul 25 02:18:34.081718 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:34.081662 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t969h\" (UniqueName: \"kubernetes.io/projected/be105f51-e248-437b-b44d-0e6303ddcd92-kube-api-access-t969h\") pod \"payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg\" (UID: \"be105f51-e248-437b-b44d-0e6303ddcd92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg" Jul 25 02:18:34.084275 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:34.084245 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e00bef-b86c-4841-a735-fce8fa46b5aa-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg\" (UID: \"a6e00bef-b86c-4841-a735-fce8fa46b5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" Jul 25 02:18:34.182294 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:34.182198 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be105f51-e248-437b-b44d-0e6303ddcd92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg\" (UID: \"be105f51-e248-437b-b44d-0e6303ddcd92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg" Jul 25 02:18:34.182294 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:34.182233 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t969h\" (UniqueName: \"kubernetes.io/projected/be105f51-e248-437b-b44d-0e6303ddcd92-kube-api-access-t969h\") pod \"payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg\" (UID: \"be105f51-e248-437b-b44d-0e6303ddcd92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg" Jul 25 02:18:34.182937 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:34.182917 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be105f51-e248-437b-b44d-0e6303ddcd92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg\" (UID: \"be105f51-e248-437b-b44d-0e6303ddcd92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg" Jul 25 02:18:34.190463 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:34.190433 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t969h\" (UniqueName: \"kubernetes.io/projected/be105f51-e248-437b-b44d-0e6303ddcd92-kube-api-access-t969h\") pod \"payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg\" (UID: \"be105f51-e248-437b-b44d-0e6303ddcd92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg" Jul 25 02:18:34.248698 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:34.248670 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg" Jul 25 02:18:34.263902 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:34.263875 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" Jul 25 02:18:34.427940 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:34.427912 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg"] Jul 25 02:18:34.429512 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:18:34.429484 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e00bef_b86c_4841_a735_fce8fa46b5aa.slice/crio-353c1c558b8ded9fa49c6171486e9002cd3c72841d43e858aa4edebca0984b95 WatchSource:0}: Error finding container 353c1c558b8ded9fa49c6171486e9002cd3c72841d43e858aa4edebca0984b95: Status 404 returned error can't find the container with id 353c1c558b8ded9fa49c6171486e9002cd3c72841d43e858aa4edebca0984b95 Jul 25 02:18:34.446578 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:34.446550 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" event={"ID":"a6e00bef-b86c-4841-a735-fce8fa46b5aa","Type":"ContainerStarted","Data":"353c1c558b8ded9fa49c6171486e9002cd3c72841d43e858aa4edebca0984b95"} Jul 25 02:18:34.606431 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:34.606404 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg"] Jul 25 02:18:34.608502 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:18:34.608473 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe105f51_e248_437b_b44d_0e6303ddcd92.slice/crio-4cff3b1d3449cd63ad0144ea6141ab3acdfc41ad42db2ff2e5becf8eb122420c WatchSource:0}: Error finding container 4cff3b1d3449cd63ad0144ea6141ab3acdfc41ad42db2ff2e5becf8eb122420c: Status 404 returned error can't find the container with id 4cff3b1d3449cd63ad0144ea6141ab3acdfc41ad42db2ff2e5becf8eb122420c Jul 25 02:18:35.452645 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:35.452604 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" event={"ID":"a6e00bef-b86c-4841-a735-fce8fa46b5aa","Type":"ContainerStarted","Data":"9cff933c8b8bc1e9116514a574bf6ca864c960e987a4f0a66d73630d74205325"} Jul 25 02:18:35.453157 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:35.452677 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" Jul 25 02:18:35.454284 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:35.454265 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg" event={"ID":"be105f51-e248-437b-b44d-0e6303ddcd92","Type":"ContainerStarted","Data":"bafe8b57e7105badb4f57641d516bce426e1d7fb729d09b2ad4cc9787e18f125"} Jul 25 02:18:35.454380 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:35.454300 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg" event={"ID":"be105f51-e248-437b-b44d-0e6303ddcd92","Type":"ContainerStarted","Data":"4cff3b1d3449cd63ad0144ea6141ab3acdfc41ad42db2ff2e5becf8eb122420c"} Jul 25 02:18:35.454420 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:35.454394 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg" Jul 25 02:18:35.472087 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:35.472037 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" podStartSLOduration=2.128369176 podStartE2EDuration="2.472018662s" totalImagesPullingTime="343.649486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:18:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:18:35.467739099 +0000 UTC m=+4056.589224948" watchObservedRunningTime="2026-07-25 02:18:35.472018662 +0000 UTC m=+4056.593504517" Jul 25 02:18:35.482754 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:35.482708 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg" podStartSLOduration=2.482694298 podStartE2EDuration="2.482694298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:18:35.481114688 +0000 UTC m=+4056.602600550" watchObservedRunningTime="2026-07-25 02:18:35.482694298 +0000 UTC m=+4056.604180171" Jul 25 02:18:41.465244 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:41.465208 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" Jul 25 02:18:46.462105 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:18:46.462076 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg" Jul 25 02:19:03.886392 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:03.886357 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg"] Jul 25 02:19:03.886947 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:03.886569 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" podUID="a6e00bef-b86c-4841-a735-fce8fa46b5aa" containerName="maas-api" containerID="cri-o://9cff933c8b8bc1e9116514a574bf6ca864c960e987a4f0a66d73630d74205325" gracePeriod=30 Jul 25 02:19:03.936959 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:03.936931 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg"] Jul 25 02:19:03.937167 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:03.937142 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg" podUID="be105f51-e248-437b-b44d-0e6303ddcd92" containerName="payload-pre-processing" containerID="cri-o://bafe8b57e7105badb4f57641d516bce426e1d7fb729d09b2ad4cc9787e18f125" gracePeriod=30 Jul 25 02:19:04.197235 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.197210 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" Jul 25 02:19:04.227774 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.227755 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg" Jul 25 02:19:04.256970 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.256943 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e00bef-b86c-4841-a735-fce8fa46b5aa-maas-api-tls\") pod \"a6e00bef-b86c-4841-a735-fce8fa46b5aa\" (UID: \"a6e00bef-b86c-4841-a735-fce8fa46b5aa\") " Jul 25 02:19:04.257125 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.256987 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be105f51-e248-437b-b44d-0e6303ddcd92-plugins-config-volume\") pod \"be105f51-e248-437b-b44d-0e6303ddcd92\" (UID: \"be105f51-e248-437b-b44d-0e6303ddcd92\") " Jul 25 02:19:04.257125 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.257031 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t969h\" (UniqueName: \"kubernetes.io/projected/be105f51-e248-437b-b44d-0e6303ddcd92-kube-api-access-t969h\") pod \"be105f51-e248-437b-b44d-0e6303ddcd92\" (UID: \"be105f51-e248-437b-b44d-0e6303ddcd92\") " Jul 25 02:19:04.257245 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.257141 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7h8b\" (UniqueName: \"kubernetes.io/projected/a6e00bef-b86c-4841-a735-fce8fa46b5aa-kube-api-access-s7h8b\") pod \"a6e00bef-b86c-4841-a735-fce8fa46b5aa\" (UID: \"a6e00bef-b86c-4841-a735-fce8fa46b5aa\") " Jul 25 02:19:04.257245 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.257167 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e00bef-b86c-4841-a735-fce8fa46b5aa-openshift-service-ca\") pod \"a6e00bef-b86c-4841-a735-fce8fa46b5aa\" (UID: \"a6e00bef-b86c-4841-a735-fce8fa46b5aa\") " Jul 25 02:19:04.257492 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.257459 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be105f51-e248-437b-b44d-0e6303ddcd92-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "be105f51-e248-437b-b44d-0e6303ddcd92" (UID: "be105f51-e248-437b-b44d-0e6303ddcd92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:19:04.257599 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.257570 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e00bef-b86c-4841-a735-fce8fa46b5aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a6e00bef-b86c-4841-a735-fce8fa46b5aa" (UID: "a6e00bef-b86c-4841-a735-fce8fa46b5aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:19:04.259213 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.259182 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e00bef-b86c-4841-a735-fce8fa46b5aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a6e00bef-b86c-4841-a735-fce8fa46b5aa" (UID: "a6e00bef-b86c-4841-a735-fce8fa46b5aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 02:19:04.259310 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.259286 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be105f51-e248-437b-b44d-0e6303ddcd92-kube-api-access-t969h" (OuterVolumeSpecName: "kube-api-access-t969h") pod "be105f51-e248-437b-b44d-0e6303ddcd92" (UID: "be105f51-e248-437b-b44d-0e6303ddcd92"). InnerVolumeSpecName "kube-api-access-t969h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:19:04.259351 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.259324 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e00bef-b86c-4841-a735-fce8fa46b5aa-kube-api-access-s7h8b" (OuterVolumeSpecName: "kube-api-access-s7h8b") pod "a6e00bef-b86c-4841-a735-fce8fa46b5aa" (UID: "a6e00bef-b86c-4841-a735-fce8fa46b5aa"). InnerVolumeSpecName "kube-api-access-s7h8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:19:04.358127 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.358096 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e00bef-b86c-4841-a735-fce8fa46b5aa-openshift-service-ca\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:19:04.358127 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.358121 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e00bef-b86c-4841-a735-fce8fa46b5aa-maas-api-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:19:04.358305 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.358135 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be105f51-e248-437b-b44d-0e6303ddcd92-plugins-config-volume\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:19:04.358305 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.358148 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t969h\" (UniqueName: \"kubernetes.io/projected/be105f51-e248-437b-b44d-0e6303ddcd92-kube-api-access-t969h\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:19:04.358305 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.358159 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7h8b\" (UniqueName: \"kubernetes.io/projected/a6e00bef-b86c-4841-a735-fce8fa46b5aa-kube-api-access-s7h8b\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:19:04.615446 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.615412 2554 generic.go:356] "Generic (PLEG): container finished" podID="a6e00bef-b86c-4841-a735-fce8fa46b5aa" containerID="9cff933c8b8bc1e9116514a574bf6ca864c960e987a4f0a66d73630d74205325" exitCode=0 Jul 25 02:19:04.615656 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.615462 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" Jul 25 02:19:04.615656 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.615512 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" event={"ID":"a6e00bef-b86c-4841-a735-fce8fa46b5aa","Type":"ContainerDied","Data":"9cff933c8b8bc1e9116514a574bf6ca864c960e987a4f0a66d73630d74205325"} Jul 25 02:19:04.615656 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.615577 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg" event={"ID":"a6e00bef-b86c-4841-a735-fce8fa46b5aa","Type":"ContainerDied","Data":"353c1c558b8ded9fa49c6171486e9002cd3c72841d43e858aa4edebca0984b95"} Jul 25 02:19:04.615656 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.615607 2554 scope.go:117] "RemoveContainer" containerID="9cff933c8b8bc1e9116514a574bf6ca864c960e987a4f0a66d73630d74205325" Jul 25 02:19:04.617096 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.617067 2554 generic.go:356] "Generic (PLEG): container finished" podID="be105f51-e248-437b-b44d-0e6303ddcd92" containerID="bafe8b57e7105badb4f57641d516bce426e1d7fb729d09b2ad4cc9787e18f125" exitCode=0 Jul 25 02:19:04.617209 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.617129 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg" event={"ID":"be105f51-e248-437b-b44d-0e6303ddcd92","Type":"ContainerDied","Data":"bafe8b57e7105badb4f57641d516bce426e1d7fb729d09b2ad4cc9787e18f125"} Jul 25 02:19:04.617209 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.617135 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg" Jul 25 02:19:04.617209 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.617169 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg" event={"ID":"be105f51-e248-437b-b44d-0e6303ddcd92","Type":"ContainerDied","Data":"4cff3b1d3449cd63ad0144ea6141ab3acdfc41ad42db2ff2e5becf8eb122420c"} Jul 25 02:19:04.627382 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.627368 2554 scope.go:117] "RemoveContainer" containerID="9cff933c8b8bc1e9116514a574bf6ca864c960e987a4f0a66d73630d74205325" Jul 25 02:19:04.627667 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:19:04.627642 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cff933c8b8bc1e9116514a574bf6ca864c960e987a4f0a66d73630d74205325\": container with ID starting with 9cff933c8b8bc1e9116514a574bf6ca864c960e987a4f0a66d73630d74205325 not found: ID does not exist" containerID="9cff933c8b8bc1e9116514a574bf6ca864c960e987a4f0a66d73630d74205325" Jul 25 02:19:04.627748 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.627679 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cff933c8b8bc1e9116514a574bf6ca864c960e987a4f0a66d73630d74205325"} err="failed to get container status \"9cff933c8b8bc1e9116514a574bf6ca864c960e987a4f0a66d73630d74205325\": rpc error: code = NotFound desc = could not find container \"9cff933c8b8bc1e9116514a574bf6ca864c960e987a4f0a66d73630d74205325\": container with ID starting with 9cff933c8b8bc1e9116514a574bf6ca864c960e987a4f0a66d73630d74205325 not found: ID does not exist" Jul 25 02:19:04.627748 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.627695 2554 scope.go:117] "RemoveContainer" containerID="bafe8b57e7105badb4f57641d516bce426e1d7fb729d09b2ad4cc9787e18f125" Jul 25 02:19:04.636028 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.636014 2554 scope.go:117] "RemoveContainer" containerID="bafe8b57e7105badb4f57641d516bce426e1d7fb729d09b2ad4cc9787e18f125" Jul 25 02:19:04.636279 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:19:04.636254 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bafe8b57e7105badb4f57641d516bce426e1d7fb729d09b2ad4cc9787e18f125\": container with ID starting with bafe8b57e7105badb4f57641d516bce426e1d7fb729d09b2ad4cc9787e18f125 not found: ID does not exist" containerID="bafe8b57e7105badb4f57641d516bce426e1d7fb729d09b2ad4cc9787e18f125" Jul 25 02:19:04.636385 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.636281 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bafe8b57e7105badb4f57641d516bce426e1d7fb729d09b2ad4cc9787e18f125"} err="failed to get container status \"bafe8b57e7105badb4f57641d516bce426e1d7fb729d09b2ad4cc9787e18f125\": rpc error: code = NotFound desc = could not find container \"bafe8b57e7105badb4f57641d516bce426e1d7fb729d09b2ad4cc9787e18f125\": container with ID starting with bafe8b57e7105badb4f57641d516bce426e1d7fb729d09b2ad4cc9787e18f125 not found: ID does not exist" Jul 25 02:19:04.637697 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.637680 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg"] Jul 25 02:19:04.641182 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.641159 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-389518-584fdf856-dnlhg"] Jul 25 02:19:04.649793 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.649767 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg"] Jul 25 02:19:04.653324 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:04.653305 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-389518-74796dc9d4-4sbtg"] Jul 25 02:19:05.381970 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:05.381935 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6e00bef-b86c-4841-a735-fce8fa46b5aa" path="/var/lib/kubelet/pods/a6e00bef-b86c-4841-a735-fce8fa46b5aa/volumes" Jul 25 02:19:05.382344 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:05.382301 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be105f51-e248-437b-b44d-0e6303ddcd92" path="/var/lib/kubelet/pods/be105f51-e248-437b-b44d-0e6303ddcd92/volumes" Jul 25 02:19:35.357320 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.357281 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw"] Jul 25 02:19:35.358005 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.357624 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" podUID="b035564b-572c-4d98-89d5-933bc4ee9749" containerName="maas-api" containerID="cri-o://71872855c3fbc3cab5b55c7ead59f35a261e35f1397044b87c337ed1d4168d97" gracePeriod=30 Jul 25 02:19:35.408539 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.408508 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b"] Jul 25 02:19:35.408726 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.408703 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b" podUID="859cced6-5143-4f87-8243-10c94bdc9a64" containerName="payload-processing" containerID="cri-o://c91bbe131fbf210262a96eb103f0a64f678eb4725a3e38ae3b4995cb5baab330" gracePeriod=30 Jul 25 02:19:35.679734 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.679708 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" Jul 25 02:19:35.710018 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.709990 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b" Jul 25 02:19:35.788502 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.788464 2554 generic.go:356] "Generic (PLEG): container finished" podID="b035564b-572c-4d98-89d5-933bc4ee9749" containerID="71872855c3fbc3cab5b55c7ead59f35a261e35f1397044b87c337ed1d4168d97" exitCode=0 Jul 25 02:19:35.788502 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.788503 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" Jul 25 02:19:35.788732 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.788546 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" event={"ID":"b035564b-572c-4d98-89d5-933bc4ee9749","Type":"ContainerDied","Data":"71872855c3fbc3cab5b55c7ead59f35a261e35f1397044b87c337ed1d4168d97"} Jul 25 02:19:35.788732 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.788628 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw" event={"ID":"b035564b-572c-4d98-89d5-933bc4ee9749","Type":"ContainerDied","Data":"39a4fbc4cedc4ecd80b1a861809f8810ebf23f84c95ffea4ef3616cc5c1452f3"} Jul 25 02:19:35.788732 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.788660 2554 scope.go:117] "RemoveContainer" containerID="71872855c3fbc3cab5b55c7ead59f35a261e35f1397044b87c337ed1d4168d97" Jul 25 02:19:35.790080 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.790049 2554 generic.go:356] "Generic (PLEG): container finished" podID="859cced6-5143-4f87-8243-10c94bdc9a64" containerID="c91bbe131fbf210262a96eb103f0a64f678eb4725a3e38ae3b4995cb5baab330" exitCode=0 Jul 25 02:19:35.790160 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.790095 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b" Jul 25 02:19:35.790160 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.790112 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b" event={"ID":"859cced6-5143-4f87-8243-10c94bdc9a64","Type":"ContainerDied","Data":"c91bbe131fbf210262a96eb103f0a64f678eb4725a3e38ae3b4995cb5baab330"} Jul 25 02:19:35.790228 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.790191 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b" event={"ID":"859cced6-5143-4f87-8243-10c94bdc9a64","Type":"ContainerDied","Data":"8962a92c814dd4b06e68e4c4a77833b303a044f6f86681e6a521880ac6ebac31"} Jul 25 02:19:35.797793 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.797775 2554 scope.go:117] "RemoveContainer" containerID="71872855c3fbc3cab5b55c7ead59f35a261e35f1397044b87c337ed1d4168d97" Jul 25 02:19:35.798080 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:19:35.798062 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71872855c3fbc3cab5b55c7ead59f35a261e35f1397044b87c337ed1d4168d97\": container with ID starting with 71872855c3fbc3cab5b55c7ead59f35a261e35f1397044b87c337ed1d4168d97 not found: ID does not exist" containerID="71872855c3fbc3cab5b55c7ead59f35a261e35f1397044b87c337ed1d4168d97" Jul 25 02:19:35.798124 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.798088 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71872855c3fbc3cab5b55c7ead59f35a261e35f1397044b87c337ed1d4168d97"} err="failed to get container status \"71872855c3fbc3cab5b55c7ead59f35a261e35f1397044b87c337ed1d4168d97\": rpc error: code = NotFound desc = could not find container \"71872855c3fbc3cab5b55c7ead59f35a261e35f1397044b87c337ed1d4168d97\": container with ID starting with 71872855c3fbc3cab5b55c7ead59f35a261e35f1397044b87c337ed1d4168d97 not found: ID does not exist" Jul 25 02:19:35.798124 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.798104 2554 scope.go:117] "RemoveContainer" containerID="c91bbe131fbf210262a96eb103f0a64f678eb4725a3e38ae3b4995cb5baab330" Jul 25 02:19:35.806132 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.806118 2554 scope.go:117] "RemoveContainer" containerID="c91bbe131fbf210262a96eb103f0a64f678eb4725a3e38ae3b4995cb5baab330" Jul 25 02:19:35.806379 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:19:35.806360 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c91bbe131fbf210262a96eb103f0a64f678eb4725a3e38ae3b4995cb5baab330\": container with ID starting with c91bbe131fbf210262a96eb103f0a64f678eb4725a3e38ae3b4995cb5baab330 not found: ID does not exist" containerID="c91bbe131fbf210262a96eb103f0a64f678eb4725a3e38ae3b4995cb5baab330" Jul 25 02:19:35.806435 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.806384 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91bbe131fbf210262a96eb103f0a64f678eb4725a3e38ae3b4995cb5baab330"} err="failed to get container status \"c91bbe131fbf210262a96eb103f0a64f678eb4725a3e38ae3b4995cb5baab330\": rpc error: code = NotFound desc = could not find container \"c91bbe131fbf210262a96eb103f0a64f678eb4725a3e38ae3b4995cb5baab330\": container with ID starting with c91bbe131fbf210262a96eb103f0a64f678eb4725a3e38ae3b4995cb5baab330 not found: ID does not exist" Jul 25 02:19:35.854640 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.854615 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgbz6\" (UniqueName: \"kubernetes.io/projected/859cced6-5143-4f87-8243-10c94bdc9a64-kube-api-access-tgbz6\") pod \"859cced6-5143-4f87-8243-10c94bdc9a64\" (UID: \"859cced6-5143-4f87-8243-10c94bdc9a64\") " Jul 25 02:19:35.854744 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.854690 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw9nw\" (UniqueName: \"kubernetes.io/projected/b035564b-572c-4d98-89d5-933bc4ee9749-kube-api-access-fw9nw\") pod \"b035564b-572c-4d98-89d5-933bc4ee9749\" (UID: \"b035564b-572c-4d98-89d5-933bc4ee9749\") " Jul 25 02:19:35.854744 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.854708 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859cced6-5143-4f87-8243-10c94bdc9a64-plugins-config-volume\") pod \"859cced6-5143-4f87-8243-10c94bdc9a64\" (UID: \"859cced6-5143-4f87-8243-10c94bdc9a64\") " Jul 25 02:19:35.854744 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.854732 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b035564b-572c-4d98-89d5-933bc4ee9749-maas-api-tls\") pod \"b035564b-572c-4d98-89d5-933bc4ee9749\" (UID: \"b035564b-572c-4d98-89d5-933bc4ee9749\") " Jul 25 02:19:35.854934 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.854758 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b035564b-572c-4d98-89d5-933bc4ee9749-openshift-service-ca\") pod \"b035564b-572c-4d98-89d5-933bc4ee9749\" (UID: \"b035564b-572c-4d98-89d5-933bc4ee9749\") " Jul 25 02:19:35.855138 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.855114 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/859cced6-5143-4f87-8243-10c94bdc9a64-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "859cced6-5143-4f87-8243-10c94bdc9a64" (UID: "859cced6-5143-4f87-8243-10c94bdc9a64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:19:35.855207 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.855188 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b035564b-572c-4d98-89d5-933bc4ee9749-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b035564b-572c-4d98-89d5-933bc4ee9749" (UID: "b035564b-572c-4d98-89d5-933bc4ee9749"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:19:35.856775 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.856757 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859cced6-5143-4f87-8243-10c94bdc9a64-kube-api-access-tgbz6" (OuterVolumeSpecName: "kube-api-access-tgbz6") pod "859cced6-5143-4f87-8243-10c94bdc9a64" (UID: "859cced6-5143-4f87-8243-10c94bdc9a64"). InnerVolumeSpecName "kube-api-access-tgbz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:19:35.856916 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.856895 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b035564b-572c-4d98-89d5-933bc4ee9749-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b035564b-572c-4d98-89d5-933bc4ee9749" (UID: "b035564b-572c-4d98-89d5-933bc4ee9749"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 02:19:35.856957 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.856940 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b035564b-572c-4d98-89d5-933bc4ee9749-kube-api-access-fw9nw" (OuterVolumeSpecName: "kube-api-access-fw9nw") pod "b035564b-572c-4d98-89d5-933bc4ee9749" (UID: "b035564b-572c-4d98-89d5-933bc4ee9749"). InnerVolumeSpecName "kube-api-access-fw9nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:19:35.956240 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.956194 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw9nw\" (UniqueName: \"kubernetes.io/projected/b035564b-572c-4d98-89d5-933bc4ee9749-kube-api-access-fw9nw\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:19:35.956240 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.956236 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859cced6-5143-4f87-8243-10c94bdc9a64-plugins-config-volume\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:19:35.956240 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.956246 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b035564b-572c-4d98-89d5-933bc4ee9749-maas-api-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:19:35.956240 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.956256 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b035564b-572c-4d98-89d5-933bc4ee9749-openshift-service-ca\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:19:35.956530 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:35.956285 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgbz6\" (UniqueName: \"kubernetes.io/projected/859cced6-5143-4f87-8243-10c94bdc9a64-kube-api-access-tgbz6\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:19:36.113105 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:36.113081 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b"] Jul 25 02:19:36.116832 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:36.116791 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-67eb46-58dbf84874-jrm4b"] Jul 25 02:19:36.127667 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:36.127648 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw"] Jul 25 02:19:36.130911 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:36.130893 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-67eb46-595b88f68c-rjzdw"] Jul 25 02:19:37.381795 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:37.381757 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="859cced6-5143-4f87-8243-10c94bdc9a64" path="/var/lib/kubelet/pods/859cced6-5143-4f87-8243-10c94bdc9a64/volumes" Jul 25 02:19:37.382333 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:19:37.382312 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b035564b-572c-4d98-89d5-933bc4ee9749" path="/var/lib/kubelet/pods/b035564b-572c-4d98-89d5-933bc4ee9749/volumes" Jul 25 02:21:01.463080 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:01.463041 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5"] Jul 25 02:21:01.463593 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:01.463272 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" podUID="91e835f5-a77c-4fa7-ac4f-0a91d4583f70" containerName="maas-api" containerID="cri-o://0dfb6f560a2d5c92edc93b2a0c74f78b99ed69d3703e6e31268119aae65e5519" gracePeriod=30 Jul 25 02:21:01.713824 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:01.713752 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" Jul 25 02:21:01.789119 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:01.789085 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91e835f5-a77c-4fa7-ac4f-0a91d4583f70-maas-api-tls\") pod \"91e835f5-a77c-4fa7-ac4f-0a91d4583f70\" (UID: \"91e835f5-a77c-4fa7-ac4f-0a91d4583f70\") " Jul 25 02:21:01.789266 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:01.789143 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72hjz\" (UniqueName: \"kubernetes.io/projected/91e835f5-a77c-4fa7-ac4f-0a91d4583f70-kube-api-access-72hjz\") pod \"91e835f5-a77c-4fa7-ac4f-0a91d4583f70\" (UID: \"91e835f5-a77c-4fa7-ac4f-0a91d4583f70\") " Jul 25 02:21:01.789266 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:01.789199 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e835f5-a77c-4fa7-ac4f-0a91d4583f70-openshift-service-ca\") pod \"91e835f5-a77c-4fa7-ac4f-0a91d4583f70\" (UID: \"91e835f5-a77c-4fa7-ac4f-0a91d4583f70\") " Jul 25 02:21:01.789576 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:01.789544 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e835f5-a77c-4fa7-ac4f-0a91d4583f70-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "91e835f5-a77c-4fa7-ac4f-0a91d4583f70" (UID: "91e835f5-a77c-4fa7-ac4f-0a91d4583f70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:21:01.791254 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:01.791227 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91e835f5-a77c-4fa7-ac4f-0a91d4583f70-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "91e835f5-a77c-4fa7-ac4f-0a91d4583f70" (UID: "91e835f5-a77c-4fa7-ac4f-0a91d4583f70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 02:21:01.791362 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:01.791308 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e835f5-a77c-4fa7-ac4f-0a91d4583f70-kube-api-access-72hjz" (OuterVolumeSpecName: "kube-api-access-72hjz") pod "91e835f5-a77c-4fa7-ac4f-0a91d4583f70" (UID: "91e835f5-a77c-4fa7-ac4f-0a91d4583f70"). InnerVolumeSpecName "kube-api-access-72hjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:21:01.890195 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:01.890164 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91e835f5-a77c-4fa7-ac4f-0a91d4583f70-maas-api-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:21:01.890195 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:01.890193 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72hjz\" (UniqueName: \"kubernetes.io/projected/91e835f5-a77c-4fa7-ac4f-0a91d4583f70-kube-api-access-72hjz\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:21:01.890381 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:01.890208 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e835f5-a77c-4fa7-ac4f-0a91d4583f70-openshift-service-ca\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:21:02.252902 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:02.252862 2554 generic.go:356] "Generic (PLEG): container finished" podID="91e835f5-a77c-4fa7-ac4f-0a91d4583f70" containerID="0dfb6f560a2d5c92edc93b2a0c74f78b99ed69d3703e6e31268119aae65e5519" exitCode=0 Jul 25 02:21:02.253066 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:02.252909 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" Jul 25 02:21:02.253066 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:02.252946 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" event={"ID":"91e835f5-a77c-4fa7-ac4f-0a91d4583f70","Type":"ContainerDied","Data":"0dfb6f560a2d5c92edc93b2a0c74f78b99ed69d3703e6e31268119aae65e5519"} Jul 25 02:21:02.253066 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:02.252998 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5" event={"ID":"91e835f5-a77c-4fa7-ac4f-0a91d4583f70","Type":"ContainerDied","Data":"1e07ca8deaa11b63d2426ff151f0097c45e48abd58b7fcea64c6fac051c427b9"} Jul 25 02:21:02.253066 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:02.253020 2554 scope.go:117] "RemoveContainer" containerID="0dfb6f560a2d5c92edc93b2a0c74f78b99ed69d3703e6e31268119aae65e5519" Jul 25 02:21:02.262642 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:02.262626 2554 scope.go:117] "RemoveContainer" containerID="0dfb6f560a2d5c92edc93b2a0c74f78b99ed69d3703e6e31268119aae65e5519" Jul 25 02:21:02.262944 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:21:02.262922 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dfb6f560a2d5c92edc93b2a0c74f78b99ed69d3703e6e31268119aae65e5519\": container with ID starting with 0dfb6f560a2d5c92edc93b2a0c74f78b99ed69d3703e6e31268119aae65e5519 not found: ID does not exist" containerID="0dfb6f560a2d5c92edc93b2a0c74f78b99ed69d3703e6e31268119aae65e5519" Jul 25 02:21:02.263047 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:02.262950 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dfb6f560a2d5c92edc93b2a0c74f78b99ed69d3703e6e31268119aae65e5519"} err="failed to get container status \"0dfb6f560a2d5c92edc93b2a0c74f78b99ed69d3703e6e31268119aae65e5519\": rpc error: code = NotFound desc = could not find container \"0dfb6f560a2d5c92edc93b2a0c74f78b99ed69d3703e6e31268119aae65e5519\": container with ID starting with 0dfb6f560a2d5c92edc93b2a0c74f78b99ed69d3703e6e31268119aae65e5519 not found: ID does not exist" Jul 25 02:21:02.275120 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:02.275094 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5"] Jul 25 02:21:02.278279 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:02.278262 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc8abe-5f9598877c-89mg5"] Jul 25 02:21:03.383315 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:03.383280 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91e835f5-a77c-4fa7-ac4f-0a91d4583f70" path="/var/lib/kubelet/pods/91e835f5-a77c-4fa7-ac4f-0a91d4583f70/volumes" Jul 25 02:21:23.411987 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.411889 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq"] Jul 25 02:21:23.412543 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.412144 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" podUID="6b6f1989-725f-4030-889b-4223aa4e084e" containerName="maas-api" containerID="cri-o://114d3e1ed01eb8f9fedc495bdf7fe2b15038f3be39ebd54ad0a8893948a7c895" gracePeriod=30 Jul 25 02:21:23.465421 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.465390 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6"] Jul 25 02:21:23.465659 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.465620 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6" podUID="9111eb02-361f-43b0-9bf1-deaf189a12b7" containerName="payload-processing" containerID="cri-o://f163f2b17731edb4aa1607eeb58d7e0874b362ff599d645fda8cde1c9a84731e" gracePeriod=30 Jul 25 02:21:23.475979 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.475946 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d"] Jul 25 02:21:23.476264 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.476217 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d" podUID="65992df1-2c85-4b51-a9ca-204b65045a38" containerName="payload-pre-processing" containerID="cri-o://52f96081cfca5c02c9b8033c814b64a5f2d0469f5b1840c813b6e565f46203f7" gracePeriod=30 Jul 25 02:21:23.791275 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.791247 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6" Jul 25 02:21:23.794168 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.794143 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" Jul 25 02:21:23.798460 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.798442 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d" Jul 25 02:21:23.886656 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.886625 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwql2\" (UniqueName: \"kubernetes.io/projected/65992df1-2c85-4b51-a9ca-204b65045a38-kube-api-access-fwql2\") pod \"65992df1-2c85-4b51-a9ca-204b65045a38\" (UID: \"65992df1-2c85-4b51-a9ca-204b65045a38\") " Jul 25 02:21:23.886834 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.886665 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7r5v\" (UniqueName: \"kubernetes.io/projected/9111eb02-361f-43b0-9bf1-deaf189a12b7-kube-api-access-p7r5v\") pod \"9111eb02-361f-43b0-9bf1-deaf189a12b7\" (UID: \"9111eb02-361f-43b0-9bf1-deaf189a12b7\") " Jul 25 02:21:23.886834 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.886685 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6f1989-725f-4030-889b-4223aa4e084e-maas-api-tls\") pod \"6b6f1989-725f-4030-889b-4223aa4e084e\" (UID: \"6b6f1989-725f-4030-889b-4223aa4e084e\") " Jul 25 02:21:23.886834 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.886711 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2w4g5\" (UniqueName: \"kubernetes.io/projected/6b6f1989-725f-4030-889b-4223aa4e084e-kube-api-access-2w4g5\") pod \"6b6f1989-725f-4030-889b-4223aa4e084e\" (UID: \"6b6f1989-725f-4030-889b-4223aa4e084e\") " Jul 25 02:21:23.886834 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.886730 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9111eb02-361f-43b0-9bf1-deaf189a12b7-plugins-config-volume\") pod \"9111eb02-361f-43b0-9bf1-deaf189a12b7\" (UID: \"9111eb02-361f-43b0-9bf1-deaf189a12b7\") " Jul 25 02:21:23.886834 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.886761 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6f1989-725f-4030-889b-4223aa4e084e-openshift-service-ca\") pod \"6b6f1989-725f-4030-889b-4223aa4e084e\" (UID: \"6b6f1989-725f-4030-889b-4223aa4e084e\") " Jul 25 02:21:23.886834 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.886828 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65992df1-2c85-4b51-a9ca-204b65045a38-plugins-config-volume\") pod \"65992df1-2c85-4b51-a9ca-204b65045a38\" (UID: \"65992df1-2c85-4b51-a9ca-204b65045a38\") " Jul 25 02:21:23.887511 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.887434 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9111eb02-361f-43b0-9bf1-deaf189a12b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9111eb02-361f-43b0-9bf1-deaf189a12b7" (UID: "9111eb02-361f-43b0-9bf1-deaf189a12b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:21:23.887713 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.887610 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6f1989-725f-4030-889b-4223aa4e084e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6b6f1989-725f-4030-889b-4223aa4e084e" (UID: "6b6f1989-725f-4030-889b-4223aa4e084e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:21:23.887713 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.887685 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65992df1-2c85-4b51-a9ca-204b65045a38-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "65992df1-2c85-4b51-a9ca-204b65045a38" (UID: "65992df1-2c85-4b51-a9ca-204b65045a38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:21:23.889111 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.889090 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6f1989-725f-4030-889b-4223aa4e084e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6b6f1989-725f-4030-889b-4223aa4e084e" (UID: "6b6f1989-725f-4030-889b-4223aa4e084e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 02:21:23.889298 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.889280 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6f1989-725f-4030-889b-4223aa4e084e-kube-api-access-2w4g5" (OuterVolumeSpecName: "kube-api-access-2w4g5") pod "6b6f1989-725f-4030-889b-4223aa4e084e" (UID: "6b6f1989-725f-4030-889b-4223aa4e084e"). InnerVolumeSpecName "kube-api-access-2w4g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:21:23.889526 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.889495 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9111eb02-361f-43b0-9bf1-deaf189a12b7-kube-api-access-p7r5v" (OuterVolumeSpecName: "kube-api-access-p7r5v") pod "9111eb02-361f-43b0-9bf1-deaf189a12b7" (UID: "9111eb02-361f-43b0-9bf1-deaf189a12b7"). InnerVolumeSpecName "kube-api-access-p7r5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:21:23.889615 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.889593 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65992df1-2c85-4b51-a9ca-204b65045a38-kube-api-access-fwql2" (OuterVolumeSpecName: "kube-api-access-fwql2") pod "65992df1-2c85-4b51-a9ca-204b65045a38" (UID: "65992df1-2c85-4b51-a9ca-204b65045a38"). InnerVolumeSpecName "kube-api-access-fwql2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:21:23.988632 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.988572 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7r5v\" (UniqueName: \"kubernetes.io/projected/9111eb02-361f-43b0-9bf1-deaf189a12b7-kube-api-access-p7r5v\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:21:23.988632 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.988596 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6f1989-725f-4030-889b-4223aa4e084e-maas-api-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:21:23.988632 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.988605 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w4g5\" (UniqueName: \"kubernetes.io/projected/6b6f1989-725f-4030-889b-4223aa4e084e-kube-api-access-2w4g5\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:21:23.988632 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.988613 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9111eb02-361f-43b0-9bf1-deaf189a12b7-plugins-config-volume\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:21:23.988632 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.988621 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6f1989-725f-4030-889b-4223aa4e084e-openshift-service-ca\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:21:23.988632 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.988630 2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65992df1-2c85-4b51-a9ca-204b65045a38-plugins-config-volume\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:21:23.988928 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:23.988637 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwql2\" (UniqueName: \"kubernetes.io/projected/65992df1-2c85-4b51-a9ca-204b65045a38-kube-api-access-fwql2\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:21:24.380316 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.380230 2554 generic.go:356] "Generic (PLEG): container finished" podID="6b6f1989-725f-4030-889b-4223aa4e084e" containerID="114d3e1ed01eb8f9fedc495bdf7fe2b15038f3be39ebd54ad0a8893948a7c895" exitCode=0 Jul 25 02:21:24.380316 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.380277 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" Jul 25 02:21:24.380539 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.380318 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" event={"ID":"6b6f1989-725f-4030-889b-4223aa4e084e","Type":"ContainerDied","Data":"114d3e1ed01eb8f9fedc495bdf7fe2b15038f3be39ebd54ad0a8893948a7c895"} Jul 25 02:21:24.380539 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.380363 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq" event={"ID":"6b6f1989-725f-4030-889b-4223aa4e084e","Type":"ContainerDied","Data":"0908303fa056da02b7b87f2bbfcc8936b1b0b5b24b8d7fdf87d457d6f83e47e9"} Jul 25 02:21:24.380539 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.380391 2554 scope.go:117] "RemoveContainer" containerID="114d3e1ed01eb8f9fedc495bdf7fe2b15038f3be39ebd54ad0a8893948a7c895" Jul 25 02:21:24.381593 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.381573 2554 generic.go:356] "Generic (PLEG): container finished" podID="65992df1-2c85-4b51-a9ca-204b65045a38" containerID="52f96081cfca5c02c9b8033c814b64a5f2d0469f5b1840c813b6e565f46203f7" exitCode=0 Jul 25 02:21:24.381707 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.381619 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d" Jul 25 02:21:24.381707 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.381664 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d" event={"ID":"65992df1-2c85-4b51-a9ca-204b65045a38","Type":"ContainerDied","Data":"52f96081cfca5c02c9b8033c814b64a5f2d0469f5b1840c813b6e565f46203f7"} Jul 25 02:21:24.381840 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.381709 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d" event={"ID":"65992df1-2c85-4b51-a9ca-204b65045a38","Type":"ContainerDied","Data":"80cac9fd8c2dc6c3121eb14d46815916721299f710c211d566b9be2e0dfea2b2"} Jul 25 02:21:24.383342 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.383149 2554 generic.go:356] "Generic (PLEG): container finished" podID="9111eb02-361f-43b0-9bf1-deaf189a12b7" containerID="f163f2b17731edb4aa1607eeb58d7e0874b362ff599d645fda8cde1c9a84731e" exitCode=0 Jul 25 02:21:24.383342 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.383204 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6" Jul 25 02:21:24.383342 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.383270 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6" event={"ID":"9111eb02-361f-43b0-9bf1-deaf189a12b7","Type":"ContainerDied","Data":"f163f2b17731edb4aa1607eeb58d7e0874b362ff599d645fda8cde1c9a84731e"} Jul 25 02:21:24.383342 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.383304 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6" event={"ID":"9111eb02-361f-43b0-9bf1-deaf189a12b7","Type":"ContainerDied","Data":"d5010021d3aa2ec16955c769f76fb36da7128e7f1a2b7b2cbfce66c1abd87dba"} Jul 25 02:21:24.392968 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.392949 2554 scope.go:117] "RemoveContainer" containerID="114d3e1ed01eb8f9fedc495bdf7fe2b15038f3be39ebd54ad0a8893948a7c895" Jul 25 02:21:24.393236 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:21:24.393206 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114d3e1ed01eb8f9fedc495bdf7fe2b15038f3be39ebd54ad0a8893948a7c895\": container with ID starting with 114d3e1ed01eb8f9fedc495bdf7fe2b15038f3be39ebd54ad0a8893948a7c895 not found: ID does not exist" containerID="114d3e1ed01eb8f9fedc495bdf7fe2b15038f3be39ebd54ad0a8893948a7c895" Jul 25 02:21:24.393333 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.393242 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114d3e1ed01eb8f9fedc495bdf7fe2b15038f3be39ebd54ad0a8893948a7c895"} err="failed to get container status \"114d3e1ed01eb8f9fedc495bdf7fe2b15038f3be39ebd54ad0a8893948a7c895\": rpc error: code = NotFound desc = could not find container \"114d3e1ed01eb8f9fedc495bdf7fe2b15038f3be39ebd54ad0a8893948a7c895\": container with ID starting with 114d3e1ed01eb8f9fedc495bdf7fe2b15038f3be39ebd54ad0a8893948a7c895 not found: ID does not exist" Jul 25 02:21:24.393333 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.393266 2554 scope.go:117] "RemoveContainer" containerID="52f96081cfca5c02c9b8033c814b64a5f2d0469f5b1840c813b6e565f46203f7" Jul 25 02:21:24.405831 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.405792 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq"] Jul 25 02:21:24.407093 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.407075 2554 scope.go:117] "RemoveContainer" containerID="52f96081cfca5c02c9b8033c814b64a5f2d0469f5b1840c813b6e565f46203f7" Jul 25 02:21:24.407359 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:21:24.407338 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f96081cfca5c02c9b8033c814b64a5f2d0469f5b1840c813b6e565f46203f7\": container with ID starting with 52f96081cfca5c02c9b8033c814b64a5f2d0469f5b1840c813b6e565f46203f7 not found: ID does not exist" containerID="52f96081cfca5c02c9b8033c814b64a5f2d0469f5b1840c813b6e565f46203f7" Jul 25 02:21:24.407423 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.407365 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f96081cfca5c02c9b8033c814b64a5f2d0469f5b1840c813b6e565f46203f7"} err="failed to get container status \"52f96081cfca5c02c9b8033c814b64a5f2d0469f5b1840c813b6e565f46203f7\": rpc error: code = NotFound desc = could not find container \"52f96081cfca5c02c9b8033c814b64a5f2d0469f5b1840c813b6e565f46203f7\": container with ID starting with 52f96081cfca5c02c9b8033c814b64a5f2d0469f5b1840c813b6e565f46203f7 not found: ID does not exist" Jul 25 02:21:24.407423 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.407382 2554 scope.go:117] "RemoveContainer" containerID="f163f2b17731edb4aa1607eeb58d7e0874b362ff599d645fda8cde1c9a84731e" Jul 25 02:21:24.408111 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.408096 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7e06ec-5b67b4b84c-m72fq"] Jul 25 02:21:24.415878 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.415628 2554 scope.go:117] "RemoveContainer" containerID="f163f2b17731edb4aa1607eeb58d7e0874b362ff599d645fda8cde1c9a84731e" Jul 25 02:21:24.415878 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:21:24.415859 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f163f2b17731edb4aa1607eeb58d7e0874b362ff599d645fda8cde1c9a84731e\": container with ID starting with f163f2b17731edb4aa1607eeb58d7e0874b362ff599d645fda8cde1c9a84731e not found: ID does not exist" containerID="f163f2b17731edb4aa1607eeb58d7e0874b362ff599d645fda8cde1c9a84731e" Jul 25 02:21:24.416229 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.415881 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f163f2b17731edb4aa1607eeb58d7e0874b362ff599d645fda8cde1c9a84731e"} err="failed to get container status \"f163f2b17731edb4aa1607eeb58d7e0874b362ff599d645fda8cde1c9a84731e\": rpc error: code = NotFound desc = could not find container \"f163f2b17731edb4aa1607eeb58d7e0874b362ff599d645fda8cde1c9a84731e\": container with ID starting with f163f2b17731edb4aa1607eeb58d7e0874b362ff599d645fda8cde1c9a84731e not found: ID does not exist" Jul 25 02:21:24.416707 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.416685 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6"] Jul 25 02:21:24.419843 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.419821 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-7e06ec-66fcb888cd-fz5j6"] Jul 25 02:21:24.428526 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.428505 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d"] Jul 25 02:21:24.433894 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:24.433874 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-7e06ec-7cd5db9dd-8hs9d"] Jul 25 02:21:25.382232 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:25.382203 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65992df1-2c85-4b51-a9ca-204b65045a38" path="/var/lib/kubelet/pods/65992df1-2c85-4b51-a9ca-204b65045a38/volumes" Jul 25 02:21:25.382577 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:25.382563 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b6f1989-725f-4030-889b-4223aa4e084e" path="/var/lib/kubelet/pods/6b6f1989-725f-4030-889b-4223aa4e084e/volumes" Jul 25 02:21:25.382890 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:25.382877 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9111eb02-361f-43b0-9bf1-deaf189a12b7" path="/var/lib/kubelet/pods/9111eb02-361f-43b0-9bf1-deaf189a12b7/volumes" Jul 25 02:21:45.889736 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:45.889703 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29"] Jul 25 02:21:45.892354 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:45.889936 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" podUID="455cb900-e4d1-4e61-b891-7d3011757d19" containerName="maas-api" containerID="cri-o://479150b9455207153dd411e8c5740a7cd953477d1cb5b63e5cfa494d10dedbba" gracePeriod=30 Jul 25 02:21:46.147306 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:46.147247 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" Jul 25 02:21:46.173749 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:46.173724 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/455cb900-e4d1-4e61-b891-7d3011757d19-maas-api-tls\") pod \"455cb900-e4d1-4e61-b891-7d3011757d19\" (UID: \"455cb900-e4d1-4e61-b891-7d3011757d19\") " Jul 25 02:21:46.173950 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:46.173756 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/455cb900-e4d1-4e61-b891-7d3011757d19-openshift-service-ca\") pod \"455cb900-e4d1-4e61-b891-7d3011757d19\" (UID: \"455cb900-e4d1-4e61-b891-7d3011757d19\") " Jul 25 02:21:46.173950 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:46.173865 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q626s\" (UniqueName: \"kubernetes.io/projected/455cb900-e4d1-4e61-b891-7d3011757d19-kube-api-access-q626s\") pod \"455cb900-e4d1-4e61-b891-7d3011757d19\" (UID: \"455cb900-e4d1-4e61-b891-7d3011757d19\") " Jul 25 02:21:46.174183 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:46.174152 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/455cb900-e4d1-4e61-b891-7d3011757d19-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "455cb900-e4d1-4e61-b891-7d3011757d19" (UID: "455cb900-e4d1-4e61-b891-7d3011757d19"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 02:21:46.176126 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:46.176088 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455cb900-e4d1-4e61-b891-7d3011757d19-kube-api-access-q626s" (OuterVolumeSpecName: "kube-api-access-q626s") pod "455cb900-e4d1-4e61-b891-7d3011757d19" (UID: "455cb900-e4d1-4e61-b891-7d3011757d19"). InnerVolumeSpecName "kube-api-access-q626s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 02:21:46.176238 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:46.176205 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/455cb900-e4d1-4e61-b891-7d3011757d19-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "455cb900-e4d1-4e61-b891-7d3011757d19" (UID: "455cb900-e4d1-4e61-b891-7d3011757d19"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 02:21:46.275015 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:46.274984 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q626s\" (UniqueName: \"kubernetes.io/projected/455cb900-e4d1-4e61-b891-7d3011757d19-kube-api-access-q626s\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:21:46.275015 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:46.275017 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/455cb900-e4d1-4e61-b891-7d3011757d19-maas-api-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:21:46.275243 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:46.275033 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/455cb900-e4d1-4e61-b891-7d3011757d19-openshift-service-ca\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\"" Jul 25 02:21:46.502644 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:46.502611 2554 generic.go:356] "Generic (PLEG): container finished" podID="455cb900-e4d1-4e61-b891-7d3011757d19" containerID="479150b9455207153dd411e8c5740a7cd953477d1cb5b63e5cfa494d10dedbba" exitCode=0 Jul 25 02:21:46.502830 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:46.502656 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" Jul 25 02:21:46.502830 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:46.502696 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" event={"ID":"455cb900-e4d1-4e61-b891-7d3011757d19","Type":"ContainerDied","Data":"479150b9455207153dd411e8c5740a7cd953477d1cb5b63e5cfa494d10dedbba"} Jul 25 02:21:46.502830 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:46.502749 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29" event={"ID":"455cb900-e4d1-4e61-b891-7d3011757d19","Type":"ContainerDied","Data":"17267ede790de5141ccaae961444f5a80d2a362a291453503cf05ff1064cbcc8"} Jul 25 02:21:46.502830 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:46.502779 2554 scope.go:117] "RemoveContainer" containerID="479150b9455207153dd411e8c5740a7cd953477d1cb5b63e5cfa494d10dedbba" Jul 25 02:21:46.512795 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:46.512778 2554 scope.go:117] "RemoveContainer" containerID="479150b9455207153dd411e8c5740a7cd953477d1cb5b63e5cfa494d10dedbba" Jul 25 02:21:46.513063 ip-10-0-135-43 kubenswrapper[2554]: E0725 02:21:46.513040 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"479150b9455207153dd411e8c5740a7cd953477d1cb5b63e5cfa494d10dedbba\": container with ID starting with 479150b9455207153dd411e8c5740a7cd953477d1cb5b63e5cfa494d10dedbba not found: ID does not exist" containerID="479150b9455207153dd411e8c5740a7cd953477d1cb5b63e5cfa494d10dedbba" Jul 25 02:21:46.513130 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:46.513075 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"479150b9455207153dd411e8c5740a7cd953477d1cb5b63e5cfa494d10dedbba"} err="failed to get container status \"479150b9455207153dd411e8c5740a7cd953477d1cb5b63e5cfa494d10dedbba\": rpc error: code = NotFound desc = could not find container \"479150b9455207153dd411e8c5740a7cd953477d1cb5b63e5cfa494d10dedbba\": container with ID starting with 479150b9455207153dd411e8c5740a7cd953477d1cb5b63e5cfa494d10dedbba not found: ID does not exist" Jul 25 02:21:46.523293 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:46.523271 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29"] Jul 25 02:21:46.525440 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:46.525417 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-575aa7-6589cfd776-87z29"] Jul 25 02:21:47.381061 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:21:47.381031 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="455cb900-e4d1-4e61-b891-7d3011757d19" path="/var/lib/kubelet/pods/455cb900-e4d1-4e61-b891-7d3011757d19/volumes" Jul 25 02:22:02.834027 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:02.833998 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-s7bpn_f8fec9f9-14dc-4a9a-a56a-169a655d827d/manager/0.log" Jul 25 02:22:02.949650 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:02.949612 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-nqfl4_6255dbe5-ec40-45da-a62e-ad6ea8fba5f9/manager/0.log" Jul 25 02:22:03.428030 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:03.428004 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9844cf678-g7bhj_b56fd0c4-c24a-4890-81a8-a56a2c2175aa/copy-manifests/0.log" Jul 25 02:22:03.434561 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:03.434541 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9844cf678-g7bhj_b56fd0c4-c24a-4890-81a8-a56a2c2175aa/manager/0.log" Jul 25 02:22:04.026292 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:04.026259 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5f8f8b9fbc-vmzj4_ceba955f-e2c5-411a-a455-b0dd171b9612/maas-api/0.log" Jul 25 02:22:04.461462 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:04.461431 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-zhxb2_8e2dfe80-c5c1-470a-832a-87088b6123a4/postgres/0.log" Jul 25 02:22:05.259342 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:05.259314 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm_ce108073-e46c-4998-80c9-b3db2a19933f/util/0.log" Jul 25 02:22:05.265786 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:05.265767 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm_ce108073-e46c-4998-80c9-b3db2a19933f/pull/0.log" Jul 25 02:22:05.271813 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:05.271778 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm_ce108073-e46c-4998-80c9-b3db2a19933f/extract/0.log" Jul 25 02:22:05.383267 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:05.383243 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn_a274c149-ae45-4bb1-9680-779b5fa4f369/util/0.log" Jul 25 02:22:05.388690 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:05.388673 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn_a274c149-ae45-4bb1-9680-779b5fa4f369/pull/0.log" Jul 25 02:22:05.393895 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:05.393879 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn_a274c149-ae45-4bb1-9680-779b5fa4f369/extract/0.log" Jul 25 02:22:05.502826 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:05.502764 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl_67b965b6-6327-4850-8bae-ce9374522219/util/0.log" Jul 25 02:22:05.508707 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:05.508676 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl_67b965b6-6327-4850-8bae-ce9374522219/pull/0.log" Jul 25 02:22:05.514583 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:05.514525 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl_67b965b6-6327-4850-8bae-ce9374522219/extract/0.log" Jul 25 02:22:05.625767 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:05.625744 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx_b8ecd12c-2b0d-42af-918e-64240ec05df6/pull/0.log" Jul 25 02:22:05.631596 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:05.631578 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx_b8ecd12c-2b0d-42af-918e-64240ec05df6/extract/0.log" Jul 25 02:22:05.636822 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:05.636789 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx_b8ecd12c-2b0d-42af-918e-64240ec05df6/util/0.log" Jul 25 02:22:05.872641 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:05.872571 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-f8lns_6492d18e-7e0c-464f-9056-69745ec53c05/manager/0.log" Jul 25 02:22:05.987308 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:05.987287 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-sgg9x_a3be6d96-aa5d-42d7-99de-1dccfafa0f88/manager/0.log" Jul 25 02:22:06.102188 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:06.102161 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-vksfp_5f2bc227-5db8-44f8-96f6-154b0c59051f/kuadrant-console-plugin/0.log" Jul 25 02:22:06.216013 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:06.215982 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-8zlzc_0f0054b1-5908-4314-b941-925a25ef9c68/registry-server/0.log" Jul 25 02:22:06.340120 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:06.340093 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-d56jb_8db6dc0d-58fa-4c88-a142-9408e7192967/manager/0.log" Jul 25 02:22:06.576587 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:06.576514 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-h9vjm_1a21652b-6337-4a5d-8205-962ad8a81b0d/manager/0.log" Jul 25 02:22:07.149288 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:07.149252 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6bd6744b-hpshk_f5b1f7e4-edc6-4178-a97e-418789d1fafd/kube-auth-proxy/0.log" Jul 25 02:22:07.634418 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:07.634385 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-cpblz_834d00ec-f37a-4e16-a9c1-c7c77d337fcf/payload-processing/0.log" Jul 25 02:22:07.743610 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:07.743580 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cdd8d58d5-6v7sn_207f125a-e453-4137-95d2-29c48898e3bf/router/0.log" Jul 25 02:22:08.089298 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:08.089269 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-g2j7v_59ef0353-b8e3-4e53-bcdd-17d6750023d1/main/0.log" Jul 25 02:22:08.438014 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:08.437989 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64f6m_b97b0365-8e1c-4c23-bf4e-b590985c6825/main/0.log" Jul 25 02:22:08.554593 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:08.554569 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-shjr6_3caf0a5e-8c56-468e-85bb-7c1581d43618/main/0.log" Jul 25 02:22:17.427689 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:17.427653 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jm7hh_62f24f02-b14f-4633-a4e0-bed308f33ada/global-pull-secret-syncer/0.log" Jul 25 02:22:17.564316 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:17.564283 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fkgvk_ac1d3e0c-04ea-4e24-ad06-91b0d2bbe0b6/konnectivity-agent/0.log" Jul 25 02:22:17.671140 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:17.671112 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-43.ec2.internal_a1bc6362a334464c48c83d833f37a3ca/haproxy/0.log" Jul 25 02:22:21.234194 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:21.234167 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm_ce108073-e46c-4998-80c9-b3db2a19933f/extract/0.log" Jul 25 02:22:21.256580 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:21.256553 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm_ce108073-e46c-4998-80c9-b3db2a19933f/util/0.log" Jul 25 02:22:21.276815 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:21.276786 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xrclm_ce108073-e46c-4998-80c9-b3db2a19933f/pull/0.log" Jul 25 02:22:21.317858 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:21.317834 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn_a274c149-ae45-4bb1-9680-779b5fa4f369/extract/0.log" Jul 25 02:22:21.352103 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:21.352079 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn_a274c149-ae45-4bb1-9680-779b5fa4f369/util/0.log" Jul 25 02:22:21.369638 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:21.369612 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p2lfn_a274c149-ae45-4bb1-9680-779b5fa4f369/pull/0.log" Jul 25 02:22:21.397259 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:21.397233 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl_67b965b6-6327-4850-8bae-ce9374522219/extract/0.log" Jul 25 02:22:21.418603 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:21.418584 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl_67b965b6-6327-4850-8bae-ce9374522219/util/0.log" Jul 25 02:22:21.439683 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:21.439662 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mkcnl_67b965b6-6327-4850-8bae-ce9374522219/pull/0.log" Jul 25 02:22:21.466319 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:21.466297 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx_b8ecd12c-2b0d-42af-918e-64240ec05df6/extract/0.log" Jul 25 02:22:21.488089 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:21.488032 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx_b8ecd12c-2b0d-42af-918e-64240ec05df6/util/0.log" Jul 25 02:22:21.509654 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:21.509634 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14c7jx_b8ecd12c-2b0d-42af-918e-64240ec05df6/pull/0.log" Jul 25 02:22:22.062940 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:22.062865 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-f8lns_6492d18e-7e0c-464f-9056-69745ec53c05/manager/0.log" Jul 25 02:22:22.096679 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:22.096653 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-sgg9x_a3be6d96-aa5d-42d7-99de-1dccfafa0f88/manager/0.log" Jul 25 02:22:22.123955 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:22.123933 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-vksfp_5f2bc227-5db8-44f8-96f6-154b0c59051f/kuadrant-console-plugin/0.log" Jul 25 02:22:22.172182 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:22.172161 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-8zlzc_0f0054b1-5908-4314-b941-925a25ef9c68/registry-server/0.log" Jul 25 02:22:22.293208 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:22.293178 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-d56jb_8db6dc0d-58fa-4c88-a142-9408e7192967/manager/0.log" Jul 25 02:22:22.468312 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:22.468281 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-h9vjm_1a21652b-6337-4a5d-8205-962ad8a81b0d/manager/0.log" Jul 25 02:22:24.271962 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:24.271929 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-fqrtv_42cfeae6-5082-45ae-a2e8-2325c0ca5ce5/cluster-monitoring-operator/0.log" Jul 25 02:22:24.604351 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:24.604279 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z7jn5_5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7/node-exporter/0.log" Jul 25 02:22:24.626575 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:24.626553 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z7jn5_5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7/kube-rbac-proxy/0.log" Jul 25 02:22:24.657634 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:24.657610 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z7jn5_5ad0b383-9ccc-41bf-aa1e-809f49f6b7c7/init-textfile/0.log" Jul 25 02:22:28.093210 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093172 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls"] Jul 25 02:22:28.093659 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093577 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b035564b-572c-4d98-89d5-933bc4ee9749" containerName="maas-api" Jul 25 02:22:28.093659 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093589 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b035564b-572c-4d98-89d5-933bc4ee9749" containerName="maas-api" Jul 25 02:22:28.093659 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093601 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be105f51-e248-437b-b44d-0e6303ddcd92" containerName="payload-pre-processing" Jul 25 02:22:28.093659 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093606 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="be105f51-e248-437b-b44d-0e6303ddcd92" containerName="payload-pre-processing" Jul 25 02:22:28.093659 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093613 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="455cb900-e4d1-4e61-b891-7d3011757d19" containerName="maas-api" Jul 25 02:22:28.093659 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093619 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="455cb900-e4d1-4e61-b891-7d3011757d19" containerName="maas-api" Jul 25 02:22:28.093659 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093626 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65992df1-2c85-4b51-a9ca-204b65045a38" containerName="payload-pre-processing" Jul 25 02:22:28.093659 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093631 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="65992df1-2c85-4b51-a9ca-204b65045a38" containerName="payload-pre-processing" Jul 25 02:22:28.093659 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093640 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9111eb02-361f-43b0-9bf1-deaf189a12b7" containerName="payload-processing" Jul 25 02:22:28.093659 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093645 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9111eb02-361f-43b0-9bf1-deaf189a12b7" containerName="payload-processing" Jul 25 02:22:28.093659 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093652 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="859cced6-5143-4f87-8243-10c94bdc9a64" containerName="payload-processing" Jul 25 02:22:28.093659 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093657 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="859cced6-5143-4f87-8243-10c94bdc9a64" containerName="payload-processing" Jul 25 02:22:28.094053 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093670 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6f1989-725f-4030-889b-4223aa4e084e" containerName="maas-api" Jul 25 02:22:28.094053 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093675 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6f1989-725f-4030-889b-4223aa4e084e" containerName="maas-api" Jul 25 02:22:28.094053 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093682 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e835f5-a77c-4fa7-ac4f-0a91d4583f70" containerName="maas-api" Jul 25 02:22:28.094053 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093687 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e835f5-a77c-4fa7-ac4f-0a91d4583f70" containerName="maas-api" Jul 25 02:22:28.094053 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093692 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6e00bef-b86c-4841-a735-fce8fa46b5aa" containerName="maas-api" Jul 25 02:22:28.094053 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093697 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6e00bef-b86c-4841-a735-fce8fa46b5aa" containerName="maas-api" Jul 25 02:22:28.094053 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093759 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6e00bef-b86c-4841-a735-fce8fa46b5aa" containerName="maas-api" Jul 25 02:22:28.094053 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093766 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e835f5-a77c-4fa7-ac4f-0a91d4583f70" containerName="maas-api" Jul 25 02:22:28.094053 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093773 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b6f1989-725f-4030-889b-4223aa4e084e" containerName="maas-api" Jul 25 02:22:28.094053 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093777 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="859cced6-5143-4f87-8243-10c94bdc9a64" containerName="payload-processing" Jul 25 02:22:28.094053 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093785 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="65992df1-2c85-4b51-a9ca-204b65045a38" containerName="payload-pre-processing" Jul 25 02:22:28.094053 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093791 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9111eb02-361f-43b0-9bf1-deaf189a12b7" containerName="payload-processing" Jul 25 02:22:28.094053 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093811 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="455cb900-e4d1-4e61-b891-7d3011757d19" containerName="maas-api" Jul 25 02:22:28.094053 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093818 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b035564b-572c-4d98-89d5-933bc4ee9749" containerName="maas-api" Jul 25 02:22:28.094053 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.093823 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="be105f51-e248-437b-b44d-0e6303ddcd92" containerName="payload-pre-processing" Jul 25 02:22:28.096932 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.096916 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" Jul 25 02:22:28.099272 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.099253 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m4dqv\"/\"kube-root-ca.crt\"" Jul 25 02:22:28.099839 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.099820 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m4dqv\"/\"openshift-service-ca.crt\"" Jul 25 02:22:28.099933 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.099901 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m4dqv\"/\"default-dockercfg-nfx4k\"" Jul 25 02:22:28.106977 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.106949 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls"] Jul 25 02:22:28.233991 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.233960 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31af13f4-ff02-43e2-82b3-006cf04d3709-proc\") pod \"perf-node-gather-daemonset-7crls\" (UID: \"31af13f4-ff02-43e2-82b3-006cf04d3709\") " pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" Jul 25 02:22:28.234141 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.233997 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31af13f4-ff02-43e2-82b3-006cf04d3709-sys\") pod \"perf-node-gather-daemonset-7crls\" (UID: \"31af13f4-ff02-43e2-82b3-006cf04d3709\") " pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" Jul 25 02:22:28.234141 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.234016 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31af13f4-ff02-43e2-82b3-006cf04d3709-podres\") pod \"perf-node-gather-daemonset-7crls\" (UID: \"31af13f4-ff02-43e2-82b3-006cf04d3709\") " pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" Jul 25 02:22:28.234141 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.234093 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vl5j\" (UniqueName: \"kubernetes.io/projected/31af13f4-ff02-43e2-82b3-006cf04d3709-kube-api-access-9vl5j\") pod \"perf-node-gather-daemonset-7crls\" (UID: \"31af13f4-ff02-43e2-82b3-006cf04d3709\") " pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" Jul 25 02:22:28.234248 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.234157 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31af13f4-ff02-43e2-82b3-006cf04d3709-lib-modules\") pod \"perf-node-gather-daemonset-7crls\" (UID: \"31af13f4-ff02-43e2-82b3-006cf04d3709\") " pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" Jul 25 02:22:28.335430 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.335395 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31af13f4-ff02-43e2-82b3-006cf04d3709-sys\") pod \"perf-node-gather-daemonset-7crls\" (UID: \"31af13f4-ff02-43e2-82b3-006cf04d3709\") " pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" Jul 25 02:22:28.335596 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.335435 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31af13f4-ff02-43e2-82b3-006cf04d3709-podres\") pod \"perf-node-gather-daemonset-7crls\" (UID: \"31af13f4-ff02-43e2-82b3-006cf04d3709\") " pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" Jul 25 02:22:28.335596 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.335463 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vl5j\" (UniqueName: \"kubernetes.io/projected/31af13f4-ff02-43e2-82b3-006cf04d3709-kube-api-access-9vl5j\") pod \"perf-node-gather-daemonset-7crls\" (UID: \"31af13f4-ff02-43e2-82b3-006cf04d3709\") " pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" Jul 25 02:22:28.335596 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.335507 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31af13f4-ff02-43e2-82b3-006cf04d3709-lib-modules\") pod \"perf-node-gather-daemonset-7crls\" (UID: \"31af13f4-ff02-43e2-82b3-006cf04d3709\") " pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" Jul 25 02:22:28.335596 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.335517 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31af13f4-ff02-43e2-82b3-006cf04d3709-sys\") pod \"perf-node-gather-daemonset-7crls\" (UID: \"31af13f4-ff02-43e2-82b3-006cf04d3709\") " pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" Jul 25 02:22:28.335596 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.335583 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31af13f4-ff02-43e2-82b3-006cf04d3709-proc\") pod \"perf-node-gather-daemonset-7crls\" (UID: \"31af13f4-ff02-43e2-82b3-006cf04d3709\") " pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" Jul 25 02:22:28.335768 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.335602 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31af13f4-ff02-43e2-82b3-006cf04d3709-podres\") pod \"perf-node-gather-daemonset-7crls\" (UID: \"31af13f4-ff02-43e2-82b3-006cf04d3709\") " pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" Jul 25 02:22:28.335768 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.335629 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31af13f4-ff02-43e2-82b3-006cf04d3709-lib-modules\") pod \"perf-node-gather-daemonset-7crls\" (UID: \"31af13f4-ff02-43e2-82b3-006cf04d3709\") " pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" Jul 25 02:22:28.335768 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.335640 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31af13f4-ff02-43e2-82b3-006cf04d3709-proc\") pod \"perf-node-gather-daemonset-7crls\" (UID: \"31af13f4-ff02-43e2-82b3-006cf04d3709\") " pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" Jul 25 02:22:28.348256 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.348195 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vl5j\" (UniqueName: \"kubernetes.io/projected/31af13f4-ff02-43e2-82b3-006cf04d3709-kube-api-access-9vl5j\") pod \"perf-node-gather-daemonset-7crls\" (UID: \"31af13f4-ff02-43e2-82b3-006cf04d3709\") " pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" Jul 25 02:22:28.407285 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.407262 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" Jul 25 02:22:28.545738 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.545704 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls"] Jul 25 02:22:28.548193 ip-10-0-135-43 kubenswrapper[2554]: W0725 02:22:28.548160 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod31af13f4_ff02_43e2_82b3_006cf04d3709.slice/crio-464e537f3625dc5e21500089733e4eeb67841501e137fdbe10673844b4ee75d6 WatchSource:0}: Error finding container 464e537f3625dc5e21500089733e4eeb67841501e137fdbe10673844b4ee75d6: Status 404 returned error can't find the container with id 464e537f3625dc5e21500089733e4eeb67841501e137fdbe10673844b4ee75d6 Jul 25 02:22:28.726418 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.726383 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" event={"ID":"31af13f4-ff02-43e2-82b3-006cf04d3709","Type":"ContainerStarted","Data":"234ddb6a23f9b3462b8a651f1b364c05558c02313b3cc9b1dd37be9832a4e495"} Jul 25 02:22:28.726567 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.726450 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" Jul 25 02:22:28.726567 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.726477 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" event={"ID":"31af13f4-ff02-43e2-82b3-006cf04d3709","Type":"ContainerStarted","Data":"464e537f3625dc5e21500089733e4eeb67841501e137fdbe10673844b4ee75d6"} Jul 25 02:22:28.751388 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:28.751345 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" podStartSLOduration=0.751330358 podStartE2EDuration="751.330358ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 02:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:22:28.749178738 +0000 UTC m=+4289.870664588" watchObservedRunningTime="2026-07-25 02:22:28.751330358 +0000 UTC m=+4289.872816222" Jul 25 02:22:29.590701 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:29.590672 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l9pbg_8f47ed40-22e9-42da-8c7a-236132c635f2/dns/0.log" Jul 25 02:22:29.610843 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:29.610820 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l9pbg_8f47ed40-22e9-42da-8c7a-236132c635f2/kube-rbac-proxy/0.log" Jul 25 02:22:29.672878 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:29.672847 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9sn77_becef38c-52d9-4baa-bfe9-30f712afabdc/dns-node-resolver/0.log" Jul 25 02:22:30.220908 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:30.220878 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d99ff7995-kcnlv_0d77b32a-87c3-4ea9-9d10-5eb70418a914/registry/0.log" Jul 25 02:22:30.285881 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:30.285856 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tb5g6_dfd8d67b-61d5-4a13-b319-3a64776c0a5b/node-ca/0.log" Jul 25 02:22:31.438287 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:31.438262 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6bd6744b-hpshk_f5b1f7e4-edc6-4178-a97e-418789d1fafd/kube-auth-proxy/0.log" Jul 25 02:22:31.622063 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:31.622014 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-cpblz_834d00ec-f37a-4e16-a9c1-c7c77d337fcf/payload-processing/0.log" Jul 25 02:22:31.649038 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:31.649015 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cdd8d58d5-6v7sn_207f125a-e453-4137-95d2-29c48898e3bf/router/0.log" Jul 25 02:22:32.459000 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:32.458973 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zc4st_a8916f19-f4e7-4bbd-a0cb-49150afe5542/serve-healthcheck-canary/0.log" Jul 25 02:22:32.949547 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:32.949524 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-lnzxj_f6485a23-898b-4080-bb1c-0f431957ff23/insights-operator/0.log" Jul 25 02:22:33.035212 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:33.035183 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gnkn9_25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b/kube-rbac-proxy/0.log" Jul 25 02:22:33.056905 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:33.056881 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gnkn9_25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b/exporter/0.log" Jul 25 02:22:33.077846 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:33.077823 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gnkn9_25beb3ea-d0d4-4d1b-bc89-b202ec6b1a7b/extractor/0.log" Jul 25 02:22:34.742470 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:34.742443 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m4dqv/perf-node-gather-daemonset-7crls" Jul 25 02:22:35.147490 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:35.147407 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-s7bpn_f8fec9f9-14dc-4a9a-a56a-169a655d827d/manager/0.log" Jul 25 02:22:35.362915 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:35.362881 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-nqfl4_6255dbe5-ec40-45da-a62e-ad6ea8fba5f9/manager/0.log" Jul 25 02:22:35.842759 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:35.842729 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9844cf678-g7bhj_b56fd0c4-c24a-4890-81a8-a56a2c2175aa/manager/0.log" Jul 25 02:22:35.865074 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:35.865050 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9844cf678-g7bhj_b56fd0c4-c24a-4890-81a8-a56a2c2175aa/copy-manifests/0.log" Jul 25 02:22:37.401121 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:37.401096 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-4k854_f01d3044-15ea-4857-a34d-f876f9bf0902/openshift-lws-operator/0.log" Jul 25 02:22:38.189184 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:38.189157 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6bd6744b-hpshk_f5b1f7e4-edc6-4178-a97e-418789d1fafd/kube-auth-proxy/0.log" Jul 25 02:22:38.331578 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:38.331550 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-cpblz_834d00ec-f37a-4e16-a9c1-c7c77d337fcf/payload-processing/0.log" Jul 25 02:22:38.344319 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:38.344286 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cdd8d58d5-6v7sn_207f125a-e453-4137-95d2-29c48898e3bf/router/0.log" Jul 25 02:22:40.015201 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:40.015166 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6bd6744b-hpshk_f5b1f7e4-edc6-4178-a97e-418789d1fafd/kube-auth-proxy/0.log" Jul 25 02:22:40.162895 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:40.162862 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-cpblz_834d00ec-f37a-4e16-a9c1-c7c77d337fcf/payload-processing/0.log" Jul 25 02:22:40.185101 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:40.185079 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cdd8d58d5-6v7sn_207f125a-e453-4137-95d2-29c48898e3bf/router/0.log" Jul 25 02:22:42.358873 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:42.358849 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-slzcl_bf1746bb-0160-47ac-81d2-4ebf692c8db1/migrator/0.log" Jul 25 02:22:42.380592 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:42.380554 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-slzcl_bf1746bb-0160-47ac-81d2-4ebf692c8db1/graceful-termination/0.log" Jul 25 02:22:43.945632 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:43.945606 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhsbs_187c72b1-1094-47c9-bb23-cbb30ab6ca4c/kube-multus-additional-cni-plugins/0.log" Jul 25 02:22:43.967325 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:43.967302 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhsbs_187c72b1-1094-47c9-bb23-cbb30ab6ca4c/egress-router-binary-copy/0.log" Jul 25 02:22:43.990491 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:43.990468 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhsbs_187c72b1-1094-47c9-bb23-cbb30ab6ca4c/cni-plugins/0.log" Jul 25 02:22:44.012893 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:44.012865 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhsbs_187c72b1-1094-47c9-bb23-cbb30ab6ca4c/bond-cni-plugin/0.log" Jul 25 02:22:44.034911 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:44.034891 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhsbs_187c72b1-1094-47c9-bb23-cbb30ab6ca4c/routeoverride-cni/0.log" Jul 25 02:22:44.055687 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:44.055661 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhsbs_187c72b1-1094-47c9-bb23-cbb30ab6ca4c/whereabouts-cni-bincopy/0.log" Jul 25 02:22:44.077400 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:44.077379 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhsbs_187c72b1-1094-47c9-bb23-cbb30ab6ca4c/whereabouts-cni/0.log" Jul 25 02:22:44.163352 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:44.163323 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cgj8z_684c7548-f719-4ab6-a653-3e95e6511d2e/kube-multus/0.log" Jul 25 02:22:44.301457 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:44.301388 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xtmst_55676d4a-8fb7-483b-b880-232cc2d10465/network-metrics-daemon/0.log" Jul 25 02:22:44.318829 ip-10-0-135-43 kubenswrapper[2554]: I0725 02:22:44.318789 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xtmst_55676d4a-8fb7-483b-b880-232cc2d10465/kube-rbac-proxy/0.log"