Jul 15 19:36:18.379328 ip-10-0-135-139 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 15 19:36:18.379342 ip-10-0-135-139 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 15 19:36:18.379352 ip-10-0-135-139 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 15 19:36:18.379717 ip-10-0-135-139 systemd[1]: Failed to start Kubernetes Kubelet. Jul 15 19:36:28.434628 ip-10-0-135-139 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 15 19:36:28.434650 ip-10-0-135-139 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 51c8fd67ca1641f8a98530b017e9abba -- Jul 15 19:38:04.254805 ip-10-0-135-139 systemd[1]: Starting Kubernetes Kubelet... Jul 15 19:38:04.711631 ip-10-0-135-139 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:38:04.711631 ip-10-0-135-139 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 19:38:04.711631 ip-10-0-135-139 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:38:04.711631 ip-10-0-135-139 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 19:38:04.711631 ip-10-0-135-139 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:38:04.712296 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.711713 2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 19:38:04.715779 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715761 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:38:04.715779 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715778 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:38:04.715937 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715782 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:38:04.715937 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715786 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:38:04.715937 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715789 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:38:04.715937 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715792 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:38:04.715937 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715795 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:38:04.715937 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715798 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:38:04.715937 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715801 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:38:04.715937 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715803 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:38:04.715937 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715806 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:38:04.715937 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715808 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:38:04.715937 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715811 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:38:04.715937 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715814 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:38:04.715937 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715816 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:38:04.715937 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715819 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:38:04.715937 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715822 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:38:04.715937 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715825 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:38:04.715937 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715833 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:38:04.715937 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715835 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:38:04.715937 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715840 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715844 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715847 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715850 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715853 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715856 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715859 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715862 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715865 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715867 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715870 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715873 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715876 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715878 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715882 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715886 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715889 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715892 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715894 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715899 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:38:04.716393 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715903 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715905 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715908 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715911 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715914 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715917 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715920 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715923 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715926 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715929 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715932 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715934 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715937 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715940 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715942 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715945 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715948 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715950 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715953 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715955 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:38:04.716881 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715958 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715961 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715963 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715965 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715968 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715972 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715975 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715979 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715982 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715984 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715987 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715990 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715992 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715994 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715997 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.715999 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716001 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716004 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716006 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716008 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:38:04.717366 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716011 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:38:04.717908 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716013 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:38:04.717908 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716016 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:38:04.717908 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716018 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:38:04.717908 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716021 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:38:04.717908 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716023 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:38:04.717908 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716026 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:38:04.717908 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716028 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:38:04.717908 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716030 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:38:04.717908 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716033 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:38:04.717908 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716035 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:38:04.717908 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716038 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:38:04.717908 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716040 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:38:04.717908 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716042 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:38:04.717908 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716045 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:38:04.717908 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716048 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:38:04.717908 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716052 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:38:04.717908 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716054 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:38:04.717908 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716057 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:38:04.717908 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716800 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716808 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716811 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716814 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716816 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716822 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716825 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716828 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716831 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716833 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716836 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716838 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716841 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716843 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716846 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716848 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716851 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716853 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716856 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716858 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:38:04.718380 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716861 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:38:04.718888 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716863 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:38:04.718888 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716865 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:38:04.718888 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716868 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:38:04.718888 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716870 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:38:04.718888 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716873 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:38:04.718888 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716875 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:38:04.718888 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716877 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:38:04.718888 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716880 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:38:04.718888 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716883 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:38:04.718888 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716888 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:38:04.718888 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716892 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:38:04.718888 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716896 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:38:04.718888 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716899 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:38:04.718888 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716902 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:38:04.718888 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716904 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:38:04.718888 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716907 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:38:04.718888 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716909 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:38:04.718888 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716912 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:38:04.718888 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716915 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716918 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716920 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716923 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716925 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716927 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716930 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716932 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716934 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716937 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716940 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716943 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716945 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716948 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716950 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716953 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716955 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716958 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716960 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716963 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:38:04.719343 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716965 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:38:04.719867 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716968 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:38:04.719867 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716970 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:38:04.719867 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716973 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:38:04.719867 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716975 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:38:04.719867 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716979 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:38:04.719867 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716981 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:38:04.719867 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716984 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:38:04.719867 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716988 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:38:04.719867 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716991 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:38:04.719867 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716994 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:38:04.719867 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.716996 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:38:04.719867 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717000 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:38:04.719867 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717003 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:38:04.719867 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717006 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:38:04.719867 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717008 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:38:04.719867 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717011 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:38:04.719867 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717014 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:38:04.719867 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717016 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:38:04.719867 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717018 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717021 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717024 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717027 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717030 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717032 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717035 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717037 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717039 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717042 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717044 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717047 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717049 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717052 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717056 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717059 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717061 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717064 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717067 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717069 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:38:04.720317 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717072 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:38:04.720810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717457 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:38:04.720810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717462 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:38:04.720810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717467 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:38:04.720810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717470 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:38:04.720810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717474 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:38:04.720810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717493 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:38:04.720810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717496 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:38:04.720810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717499 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:38:04.720810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717502 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:38:04.720810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717505 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:38:04.720810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717507 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:38:04.720810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717510 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:38:04.720810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717512 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:38:04.720810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717515 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:38:04.720810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717518 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:38:04.720810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717520 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:38:04.720810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717523 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:38:04.720810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717526 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:38:04.720810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717528 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717530 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717533 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717535 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717538 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717540 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717543 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717546 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717549 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717551 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717554 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717557 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717559 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717561 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717564 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717566 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717569 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717572 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717574 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717577 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:38:04.721277 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717580 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717582 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717585 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717587 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717589 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717592 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717595 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717597 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717599 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717602 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717604 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717607 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717609 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717611 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717614 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717616 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717618 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717621 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717623 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717626 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:38:04.721775 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717629 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717631 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717635 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717638 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717641 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717643 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717645 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717648 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717652 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717655 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717658 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717660 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717663 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717666 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717668 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717671 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717673 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717675 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717678 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717680 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:38:04.722260 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717683 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:38:04.722783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717685 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:38:04.722783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717688 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:38:04.722783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717690 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:38:04.722783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717692 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:38:04.722783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717695 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:38:04.722783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717697 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:38:04.722783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717700 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:38:04.722783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717702 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:38:04.722783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717705 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:38:04.722783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717707 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:38:04.722783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717710 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:38:04.722783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717713 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:38:04.722783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717715 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:38:04.722783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717719 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:38:04.722783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717722 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:38:04.722783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717724 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:38:04.722783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717727 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:38:04.722783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717729 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:38:04.722783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717733 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.717736 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717809 2554 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717816 2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717823 2554 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717827 2554 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717832 2554 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717835 2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717839 2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717843 2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717846 2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717849 2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717853 2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717856 2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717859 2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717862 2554 flags.go:64] FLAG: --cgroup-root="" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717865 2554 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717868 2554 flags.go:64] FLAG: --client-ca-file="" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717871 2554 flags.go:64] FLAG: --cloud-provider="external" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717873 2554 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717878 2554 flags.go:64] FLAG: --cluster-domain="" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717881 2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717884 2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717887 2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 19:38:04.723234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717891 2554 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717894 2554 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717897 2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717901 2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717905 2554 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717908 2554 flags.go:64] FLAG: --contention-profiling="false" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717911 2554 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717914 2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717917 2554 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717920 2554 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717924 2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717927 2554 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717930 2554 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717933 2554 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717936 2554 flags.go:64] FLAG: --enable-server="true" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717939 2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717944 2554 flags.go:64] FLAG: --event-burst="100" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717947 2554 flags.go:64] FLAG: --event-qps="50" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717949 2554 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717952 2554 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717955 2554 flags.go:64] FLAG: --eviction-hard="" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717958 2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717961 2554 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717964 2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717967 2554 flags.go:64] FLAG: --eviction-soft="" Jul 15 19:38:04.723848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717969 2554 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717972 2554 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717975 2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717978 2554 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717981 2554 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717983 2554 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717986 2554 flags.go:64] FLAG: --feature-gates="" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717991 2554 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717994 2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.717997 2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718001 2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718004 2554 flags.go:64] FLAG: --healthz-port="10248" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718007 2554 flags.go:64] FLAG: --help="false" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718010 2554 flags.go:64] FLAG: --hostname-override="ip-10-0-135-139.ec2.internal" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718013 2554 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718016 2554 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718018 2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718022 2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718025 2554 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718028 2554 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718031 2554 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718033 2554 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718036 2554 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 19:38:04.724435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718039 2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718043 2554 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718045 2554 flags.go:64] FLAG: --kube-reserved="" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718048 2554 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718051 2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718054 2554 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718057 2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718060 2554 flags.go:64] FLAG: --lock-file="" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718062 2554 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718065 2554 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718068 2554 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718073 2554 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718076 2554 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718079 2554 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718081 2554 flags.go:64] FLAG: --logging-format="text" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718084 2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718089 2554 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718091 2554 flags.go:64] FLAG: --manifest-url="" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718094 2554 flags.go:64] FLAG: --manifest-url-header="" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718099 2554 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718102 2554 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718106 2554 flags.go:64] FLAG: --max-pods="110" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718109 2554 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718112 2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718115 2554 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 19:38:04.725042 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718118 2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718120 2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718123 2554 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718126 2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718133 2554 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718136 2554 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718139 2554 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718143 2554 flags.go:64] FLAG: --pod-cidr="" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718146 2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718152 2554 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718155 2554 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718158 2554 flags.go:64] FLAG: --pods-per-core="0" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718161 2554 flags.go:64] FLAG: --port="10250" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718164 2554 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718167 2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ecd0e60749ae2f0a" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718170 2554 flags.go:64] FLAG: --qos-reserved="" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718172 2554 flags.go:64] FLAG: --read-only-port="10255" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718176 2554 flags.go:64] FLAG: --register-node="true" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718178 2554 flags.go:64] FLAG: --register-with-taints="" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718182 2554 flags.go:64] FLAG: --registry-burst="10" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718185 2554 flags.go:64] FLAG: --registry-qps="5" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718187 2554 flags.go:64] FLAG: --reserved-cpus="" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718190 2554 flags.go:64] FLAG: --reserved-memory="" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718193 2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718197 2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 19:38:04.725718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718200 2554 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718204 2554 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718207 2554 flags.go:64] FLAG: --runonce="false" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718210 2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718213 2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718216 2554 flags.go:64] FLAG: --seccomp-default="false" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718219 2554 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718221 2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718224 2554 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718227 2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718230 2554 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718233 2554 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718235 2554 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718238 2554 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718241 2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718244 2554 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718247 2554 flags.go:64] FLAG: --system-cgroups="" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718250 2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718255 2554 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718257 2554 flags.go:64] FLAG: --tls-cert-file="" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718260 2554 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718265 2554 flags.go:64] FLAG: --tls-min-version="" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718267 2554 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718270 2554 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718273 2554 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 19:38:04.726352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718275 2554 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718278 2554 flags.go:64] FLAG: --v="2" Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718282 2554 flags.go:64] FLAG: --version="false" Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718286 2554 flags.go:64] FLAG: --vmodule="" Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718290 2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.718293 2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718406 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718410 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718415 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718417 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718420 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718423 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718425 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718428 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718430 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718433 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718435 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718438 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718440 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718443 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718445 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:38:04.726972 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718448 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:38:04.727491 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718451 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:38:04.727491 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718454 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:38:04.727491 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718459 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:38:04.727491 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718463 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:38:04.727491 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718465 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:38:04.727491 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718468 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:38:04.727491 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718470 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:38:04.727491 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718473 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:38:04.727491 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718475 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:38:04.727491 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718491 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:38:04.727491 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718494 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:38:04.727491 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718497 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:38:04.727491 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718499 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:38:04.727491 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718502 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:38:04.727491 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718505 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:38:04.727491 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718507 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:38:04.727491 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718511 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:38:04.727491 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718514 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:38:04.727491 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718518 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718521 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718523 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718526 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718528 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718531 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718533 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718536 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718538 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718541 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718543 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718546 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718548 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718550 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718553 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718555 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718557 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718561 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718563 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718565 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:38:04.727953 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718568 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:38:04.728439 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718570 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:38:04.728439 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718573 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:38:04.728439 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718575 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:38:04.728439 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718577 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:38:04.728439 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718580 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:38:04.728439 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718583 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:38:04.728439 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718585 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:38:04.728439 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718587 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:38:04.728439 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718590 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:38:04.728439 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718594 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:38:04.728439 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718596 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:38:04.728439 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718601 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:38:04.728439 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718604 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:38:04.728439 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718607 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:38:04.728439 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718611 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:38:04.728439 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718614 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:38:04.728439 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718618 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:38:04.728439 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718620 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:38:04.728439 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718623 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718626 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718629 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718632 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718634 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718638 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718641 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718644 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718646 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718649 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718652 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718655 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718657 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718660 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718662 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718665 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718667 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718670 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718672 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718675 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:38:04.728997 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718677 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:38:04.729466 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718679 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:38:04.729466 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718682 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:38:04.729466 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718686 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:38:04.729466 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718688 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:38:04.729466 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.718692 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:38:04.729466 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.719412 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:38:04.729466 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.727518 2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 19:38:04.729466 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.727545 2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 19:38:04.729466 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.728389 2554 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 19:38:04.729466 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.728396 2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 19:38:04.729466 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728477 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:38:04.729466 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728500 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:38:04.729466 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728503 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:38:04.729466 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728507 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:38:04.729466 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728509 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728513 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728516 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728519 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728522 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728524 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728527 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728529 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728532 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728534 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728537 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728539 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728541 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728545 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728548 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728550 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728553 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728556 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728559 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728561 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:38:04.729856 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728564 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728566 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728569 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728571 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728574 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728583 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728586 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728588 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728590 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728593 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728595 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728598 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728600 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728603 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728605 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728607 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728610 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728612 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728615 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728617 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:38:04.730339 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728620 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:38:04.730839 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728622 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:38:04.730839 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728625 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:38:04.730839 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728627 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:38:04.730839 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728631 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:38:04.730839 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728633 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:38:04.730839 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728636 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:38:04.730839 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728638 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:38:04.730839 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728641 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:38:04.730839 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728643 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:38:04.730839 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728646 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:38:04.730839 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728648 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:38:04.730839 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728652 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:38:04.730839 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728656 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:38:04.730839 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728659 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:38:04.730839 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728662 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:38:04.730839 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728665 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:38:04.730839 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728667 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:38:04.730839 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728677 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:38:04.731265 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728680 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:38:04.731265 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728684 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:38:04.731265 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728686 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:38:04.731265 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728696 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:38:04.731265 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728700 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:38:04.731265 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728703 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:38:04.731265 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728705 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:38:04.731265 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728708 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:38:04.731265 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728710 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:38:04.731265 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728713 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:38:04.731265 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728715 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:38:04.731265 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728718 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:38:04.731265 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728720 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:38:04.731265 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728723 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:38:04.731265 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728725 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:38:04.731265 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728727 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:38:04.731265 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728730 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:38:04.731265 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728732 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:38:04.731265 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728735 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:38:04.731740 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728738 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:38:04.731740 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728740 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:38:04.731740 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728742 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:38:04.731740 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728745 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:38:04.731740 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728748 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:38:04.731740 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728750 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:38:04.731740 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728753 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:38:04.731740 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728755 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:38:04.731740 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728757 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:38:04.731740 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728760 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:38:04.731740 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728763 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:38:04.731740 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728766 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:38:04.731740 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728768 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:38:04.731740 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728776 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:38:04.731740 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728779 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:38:04.731740 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728781 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:38:04.731740 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728784 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:38:04.732193 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.728789 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:38:04.732193 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728980 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:38:04.732193 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728985 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:38:04.732193 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728989 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:38:04.732193 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728993 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:38:04.732193 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.728997 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:38:04.732193 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729000 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:38:04.732193 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729003 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:38:04.732193 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729006 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:38:04.732193 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729009 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:38:04.732193 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729012 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:38:04.732193 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729015 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:38:04.732193 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729018 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:38:04.732193 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729021 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:38:04.732193 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729024 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:38:04.732572 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729026 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:38:04.732572 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729029 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:38:04.732572 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729031 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:38:04.732572 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729034 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:38:04.732572 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729036 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:38:04.732572 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729039 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:38:04.732572 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729041 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:38:04.732572 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729044 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:38:04.732572 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729046 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:38:04.732572 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729049 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:38:04.732572 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729051 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:38:04.732572 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729054 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:38:04.732572 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729056 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:38:04.732572 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729059 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:38:04.732572 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729067 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:38:04.732572 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729070 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:38:04.732572 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729072 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:38:04.732572 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729075 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:38:04.732572 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729079 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:38:04.733051 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729082 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:38:04.733051 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729084 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:38:04.733051 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729086 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:38:04.733051 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729089 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:38:04.733051 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729091 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:38:04.733051 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729094 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:38:04.733051 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729096 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:38:04.733051 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729098 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:38:04.733051 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729101 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:38:04.733051 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729103 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:38:04.733051 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729106 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:38:04.733051 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729109 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:38:04.733051 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729111 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:38:04.733051 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729114 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:38:04.733051 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729118 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:38:04.733051 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729121 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:38:04.733051 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729124 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:38:04.733051 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729126 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:38:04.733051 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729129 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729131 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729134 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729136 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729139 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729141 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729143 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729146 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729148 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729151 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729161 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729163 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729166 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729168 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729170 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729173 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729175 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729178 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729180 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729183 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:38:04.733509 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729185 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729188 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729190 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729192 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729195 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729197 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729200 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729202 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729205 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729207 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729210 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729212 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729215 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729217 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729220 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729223 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729225 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729228 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729230 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729232 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:38:04.734050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729235 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:38:04.734561 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729237 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:38:04.734561 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729239 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:38:04.734561 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729250 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:38:04.734561 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729253 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:38:04.734561 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729255 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:38:04.734561 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729258 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:38:04.734561 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:04.729260 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:38:04.734561 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.729265 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:38:04.734561 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.729395 2554 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 19:38:04.734561 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.731842 2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 19:38:04.734561 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.733587 2554 server.go:1019] "Starting client certificate rotation" Jul 15 19:38:04.734561 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.733712 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 19:38:04.734561 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.734542 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 19:38:04.736576 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.736547 2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 19:38:04.740879 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.740847 2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 19:38:04.752990 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.752966 2554 log.go:25] "Validated CRI v1 runtime API" Jul 15 19:38:04.756352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.756334 2554 log.go:25] "Validated CRI v1 image API" Jul 15 19:38:04.757195 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.757179 2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 19:38:04.759798 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.759780 2554 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8a756e07-e63c-4683-8aa4-5137dd2daa5d:/dev/nvme0n1p4 9323ccd3-1edb-49a1-bc00-11f60ff94e8a:/dev/nvme0n1p3] Jul 15 19:38:04.759852 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.759799 2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 19:38:04.764225 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.764098 2554 manager.go:216] Machine: {Timestamp:2026-07-15 19:38:04.763622687 +0000 UTC m=+0.373399138 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100240 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26006a3178217d08766e31fee5b2b7 SystemUUID:ec26006a-3178-217d-0876-6e31fee5b2b7 BootID:51c8fd67-ca16-41f8-a985-30b017e9abba Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:65:ef:6e:09:a9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:65:ef:6e:09:a9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:aa:b2:0c:b5:11 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 19:38:04.764225 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.764222 2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 19:38:04.764346 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.764299 2554 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 19:38:04.768036 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.768007 2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 19:38:04.768172 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.768041 2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-139.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 19:38:04.769016 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.769007 2554 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 19:38:04.769051 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.769021 2554 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 19:38:04.769051 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.769032 2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 19:38:04.769051 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.769046 2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 19:38:04.769127 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.769111 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 19:38:04.770496 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.770473 2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 19:38:04.774134 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.774123 2554 state_mem.go:36] "Initialized new in-memory state store" Jul 15 19:38:04.774245 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.774236 2554 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 19:38:04.774274 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.774264 2554 kubelet.go:486] "Attempting to sync node with API server" Jul 15 19:38:04.774305 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.774276 2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 19:38:04.774305 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.774289 2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 19:38:04.774305 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.774298 2554 kubelet.go:398] "Adding apiserver pod source" Jul 15 19:38:04.774382 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.774308 2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 19:38:04.774701 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.774687 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 19:38:04.774772 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.774712 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 19:38:04.777406 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.777392 2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 19:38:04.779157 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.779143 2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 19:38:04.779197 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.779174 2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 19:38:04.780630 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.780618 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 19:38:04.780684 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.780636 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 19:38:04.780684 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.780643 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 19:38:04.780684 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.780648 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 19:38:04.780684 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.780658 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 19:38:04.780684 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.780664 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 19:38:04.780684 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.780672 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 19:38:04.780684 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.780677 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 19:38:04.780684 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.780685 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 19:38:04.780886 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.780690 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 19:38:04.780886 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.780698 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 19:38:04.780886 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.780704 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 19:38:04.782733 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.782690 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 19:38:04.782878 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.782822 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 19:38:04.786810 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.786794 2554 server.go:1268] "Started kubelet" Jul 15 19:38:04.786940 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.786909 2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 19:38:04.786940 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.786898 2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 19:38:04.787036 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.786970 2554 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 19:38:04.787450 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.787421 2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-139.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 19:38:04.787450 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:04.787433 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 19:38:04.787602 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:04.787496 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 19:38:04.787773 ip-10-0-135-139 systemd[1]: Started Kubernetes Kubelet. Jul 15 19:38:04.788363 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.788343 2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 19:38:04.790036 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.790022 2554 server.go:310] "Adding debug handlers to kubelet server" Jul 15 19:38:04.793947 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.793928 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 19:38:04.794063 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.793932 2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 19:38:04.794134 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:04.794042 2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 19:38:04.794247 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:04.794225 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-139.ec2.internal\" not found" Jul 15 19:38:04.794835 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.794815 2554 factory.go:55] Registering systemd factory Jul 15 19:38:04.794933 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.794859 2554 factory.go:223] Registration of the systemd container factory successfully Jul 15 19:38:04.794933 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.794818 2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 19:38:04.794933 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.794821 2554 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 19:38:04.794933 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.794910 2554 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 19:38:04.795091 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.794974 2554 reconstruct.go:97] "Volume reconstruction finished" Jul 15 19:38:04.795091 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.794983 2554 reconciler.go:29] "Reconciler: start to sync state" Jul 15 19:38:04.795091 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.795038 2554 factory.go:153] Registering CRI-O factory Jul 15 19:38:04.795091 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.795049 2554 factory.go:223] Registration of the crio container factory successfully Jul 15 19:38:04.795091 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.795084 2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 19:38:04.795245 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.795101 2554 factory.go:103] Registering Raw factory Jul 15 19:38:04.795245 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.795111 2554 manager.go:1196] Started watching for new ooms in manager Jul 15 19:38:04.795591 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.795577 2554 manager.go:349] Starting recovery of all containers Jul 15 19:38:04.802285 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:04.802242 2554 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 19:38:04.802285 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:04.802277 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 19:38:04.802910 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:04.801804 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-139.ec2.internal.18c28dc632c22924 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-139.ec2.internal,UID:ip-10-0-135-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-139.ec2.internal,},FirstTimestamp:2026-07-15 19:38:04.786764068 +0000 UTC m=+0.396540509,LastTimestamp:2026-07-15 19:38:04.786764068 +0000 UTC m=+0.396540509,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-139.ec2.internal,}" Jul 15 19:38:04.806459 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.806348 2554 manager.go:354] Recovery completed Jul 15 19:38:04.809226 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:04.809204 2554 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 15 19:38:04.812679 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.812666 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:38:04.814590 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.814566 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:38:04.814650 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.814603 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:38:04.814650 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.814613 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:38:04.815125 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.815109 2554 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 19:38:04.815125 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.815124 2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 19:38:04.815285 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.815141 2554 state_mem.go:36] "Initialized new in-memory state store" Jul 15 19:38:04.815285 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.815164 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:38:04.815863 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.815849 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:38:04.815937 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.815871 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:38:04.815937 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.815885 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:38:04.816475 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:04.816406 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-139.ec2.internal.18c28dc6346abfe7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-139.ec2.internal,UID:ip-10-0-135-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-139.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-139.ec2.internal,},FirstTimestamp:2026-07-15 19:38:04.814589927 +0000 UTC m=+0.424366368,LastTimestamp:2026-07-15 19:38:04.814589927 +0000 UTC m=+0.424366368,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-139.ec2.internal,}" Jul 15 19:38:04.816475 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.816500 2554 policy_none.go:49] "None policy: Start" Jul 15 19:38:04.816475 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.816515 2554 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 19:38:04.816740 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.816528 2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 19:38:04.817471 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.817458 2554 policy_none.go:47] "Start" Jul 15 19:38:04.825430 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:04.825363 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-139.ec2.internal.18c28dc6346b04f4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-139.ec2.internal,UID:ip-10-0-135-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-139.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-139.ec2.internal,},FirstTimestamp:2026-07-15 19:38:04.814607604 +0000 UTC m=+0.424384044,LastTimestamp:2026-07-15 19:38:04.814607604 +0000 UTC m=+0.424384044,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-139.ec2.internal,}" Jul 15 19:38:04.827629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.827611 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n9qd2" Jul 15 19:38:04.835938 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.835918 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n9qd2" Jul 15 19:38:04.835938 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:04.835856 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-139.ec2.internal.18c28dc6346b2a8a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-139.ec2.internal,UID:ip-10-0-135-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-139.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-139.ec2.internal,},FirstTimestamp:2026-07-15 19:38:04.814617226 +0000 UTC m=+0.424393668,LastTimestamp:2026-07-15 19:38:04.814617226 +0000 UTC m=+0.424393668,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-139.ec2.internal,}" Jul 15 19:38:04.856111 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.856096 2554 manager.go:337] "Starting Device Plugin manager" Jul 15 19:38:04.864434 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:04.856128 2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 19:38:04.864434 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.856137 2554 server.go:87] "Starting device plugin registration server" Jul 15 19:38:04.864434 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.856387 2554 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 19:38:04.864434 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.856402 2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 19:38:04.864434 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.856540 2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 19:38:04.864434 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.856614 2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 19:38:04.864434 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.856620 2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 19:38:04.864434 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:04.857126 2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 19:38:04.864434 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:04.857181 2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-139.ec2.internal\" not found" Jul 15 19:38:04.865450 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.865423 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 19:38:04.866794 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.866780 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 19:38:04.866871 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.866814 2554 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 19:38:04.866871 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.866831 2554 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 19:38:04.866871 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:04.866859 2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 19:38:04.868658 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.868638 2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:38:04.957504 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.957451 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:38:04.958697 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.958676 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:38:04.958817 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.958718 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:38:04.958817 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.958732 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:38:04.958817 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.958769 2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-139.ec2.internal" Jul 15 19:38:04.967432 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.967377 2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-139.ec2.internal"] Jul 15 19:38:04.967560 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.967448 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:38:04.968656 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.968629 2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-139.ec2.internal" Jul 15 19:38:04.968753 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:04.968665 2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-139.ec2.internal\": node \"ip-10-0-135-139.ec2.internal\" not found" Jul 15 19:38:04.969117 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.969101 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:38:04.969187 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.969135 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:38:04.969187 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.969149 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:38:04.970443 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.970428 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:38:04.970558 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.970544 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal" Jul 15 19:38:04.970608 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.970584 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:38:04.971170 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.971152 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:38:04.971250 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.971154 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:38:04.971250 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.971190 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:38:04.971250 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.971206 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:38:04.971250 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.971214 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:38:04.971250 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.971231 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:38:04.972591 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.972576 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-139.ec2.internal" Jul 15 19:38:04.972679 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.972607 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:38:04.973265 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.973253 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:38:04.973331 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.973276 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:38:04.973331 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:04.973287 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:38:04.991431 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:04.991410 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-139.ec2.internal\" not found" Jul 15 19:38:05.001410 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:05.001393 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-139.ec2.internal\" not found" node="ip-10-0-135-139.ec2.internal" Jul 15 19:38:05.006052 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:05.006035 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-139.ec2.internal\" not found" node="ip-10-0-135-139.ec2.internal" Jul 15 19:38:05.092240 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:05.092211 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-139.ec2.internal\" not found" Jul 15 19:38:05.096520 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.096506 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ff7075f5bca7a3a4c83421a497bc75eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal\" (UID: \"ff7075f5bca7a3a4c83421a497bc75eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal" Jul 15 19:38:05.096585 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.096530 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff7075f5bca7a3a4c83421a497bc75eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal\" (UID: \"ff7075f5bca7a3a4c83421a497bc75eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal" Jul 15 19:38:05.096585 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.096548 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4dd0763a6110d8ba2bd7fcafeff6d0d7-config\") pod \"kube-apiserver-proxy-ip-10-0-135-139.ec2.internal\" (UID: \"4dd0763a6110d8ba2bd7fcafeff6d0d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-139.ec2.internal" Jul 15 19:38:05.192558 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:05.192519 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-139.ec2.internal\" not found" Jul 15 19:38:05.196856 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.196836 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ff7075f5bca7a3a4c83421a497bc75eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal\" (UID: \"ff7075f5bca7a3a4c83421a497bc75eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal" Jul 15 19:38:05.196910 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.196868 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff7075f5bca7a3a4c83421a497bc75eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal\" (UID: \"ff7075f5bca7a3a4c83421a497bc75eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal" Jul 15 19:38:05.196910 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.196896 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4dd0763a6110d8ba2bd7fcafeff6d0d7-config\") pod \"kube-apiserver-proxy-ip-10-0-135-139.ec2.internal\" (UID: \"4dd0763a6110d8ba2bd7fcafeff6d0d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-139.ec2.internal" Jul 15 19:38:05.196984 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.196943 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4dd0763a6110d8ba2bd7fcafeff6d0d7-config\") pod \"kube-apiserver-proxy-ip-10-0-135-139.ec2.internal\" (UID: \"4dd0763a6110d8ba2bd7fcafeff6d0d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-139.ec2.internal" Jul 15 19:38:05.196984 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.196950 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff7075f5bca7a3a4c83421a497bc75eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal\" (UID: \"ff7075f5bca7a3a4c83421a497bc75eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal" Jul 15 19:38:05.196984 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.196943 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ff7075f5bca7a3a4c83421a497bc75eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal\" (UID: \"ff7075f5bca7a3a4c83421a497bc75eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal" Jul 15 19:38:05.293376 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:05.293295 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-139.ec2.internal\" not found" Jul 15 19:38:05.304786 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.304764 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal" Jul 15 19:38:05.309957 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.309940 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-139.ec2.internal" Jul 15 19:38:05.394029 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:05.393991 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-139.ec2.internal\" not found" Jul 15 19:38:05.494488 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:05.494445 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-139.ec2.internal\" not found" Jul 15 19:38:05.594977 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:05.594898 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-139.ec2.internal\" not found" Jul 15 19:38:05.695300 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:05.695268 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-139.ec2.internal\" not found" Jul 15 19:38:05.732634 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.732608 2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 19:38:05.733249 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.732770 2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 19:38:05.795681 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:05.795618 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-139.ec2.internal\" not found" Jul 15 19:38:05.806566 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.806544 2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:38:05.807942 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.807921 2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:38:05.816121 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:05.816093 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7075f5bca7a3a4c83421a497bc75eb.slice/crio-0bf67c2d162694d7fb8681f6ad0c0de834403ee601e431d4a6ceb89ffb1ec583 WatchSource:0}: Error finding container 0bf67c2d162694d7fb8681f6ad0c0de834403ee601e431d4a6ceb89ffb1ec583: Status 404 returned error can't find the container with id 0bf67c2d162694d7fb8681f6ad0c0de834403ee601e431d4a6ceb89ffb1ec583 Jul 15 19:38:05.816300 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:05.816283 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd0763a6110d8ba2bd7fcafeff6d0d7.slice/crio-f3b3076913b5de74580fcfb79cc08b7b4efb0a28fb7e79354dd95e070988e131 WatchSource:0}: Error finding container f3b3076913b5de74580fcfb79cc08b7b4efb0a28fb7e79354dd95e070988e131: Status 404 returned error can't find the container with id f3b3076913b5de74580fcfb79cc08b7b4efb0a28fb7e79354dd95e070988e131 Jul 15 19:38:05.822900 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.822880 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:38:05.840549 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.840513 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 19:33:06 +0000 UTC" deadline="2028-01-13 17:39:16.670217038 +0000 UTC" Jul 15 19:38:05.840549 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.840542 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13126h1m10.829679838s" Jul 15 19:38:05.870053 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.869945 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal" event={"ID":"ff7075f5bca7a3a4c83421a497bc75eb","Type":"ContainerStarted","Data":"0bf67c2d162694d7fb8681f6ad0c0de834403ee601e431d4a6ceb89ffb1ec583"} Jul 15 19:38:05.871023 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.871001 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-139.ec2.internal" event={"ID":"4dd0763a6110d8ba2bd7fcafeff6d0d7","Type":"ContainerStarted","Data":"f3b3076913b5de74580fcfb79cc08b7b4efb0a28fb7e79354dd95e070988e131"} Jul 15 19:38:05.895226 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.895194 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal" Jul 15 19:38:05.911684 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.911656 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 19:38:05.914371 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.914354 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-139.ec2.internal" Jul 15 19:38:05.927221 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.927197 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 19:38:05.969423 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:05.969392 2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:38:08.141032 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.141001 2554 apiserver.go:52] "Watching apiserver" Jul 15 19:38:08.150592 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.150552 2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 19:38:08.151045 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.151017 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-mhtt6","kube-system/kube-apiserver-proxy-ip-10-0-135-139.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx","openshift-dns/node-resolver-zswzk","openshift-image-registry/node-ca-wxt6l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal","openshift-multus/network-metrics-daemon-477vh","openshift-network-operator/iptables-alerter-zsnk8","kube-system/global-pull-secret-syncer-rv9rh","openshift-cluster-node-tuning-operator/tuned-487tv","openshift-multus/multus-additional-cni-plugins-9mp58","openshift-multus/multus-n9dx5","openshift-network-diagnostics/network-check-target-2ws6h","openshift-ovn-kubernetes/ovnkube-node-nxd2f"] Jul 15 19:38:08.152834 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.152810 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:08.152943 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:08.152915 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-477vh" podUID="33e0c74e-e752-4c0f-9c26-a9742b866f41" Jul 15 19:38:08.153941 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.153744 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:08.153941 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:08.153834 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ws6h" podUID="9db8df3b-ca71-41fb-9039-a9b1afc9f1bd" Jul 15 19:38:08.154945 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.154920 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.156311 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.156294 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zswzk" Jul 15 19:38:08.157229 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.157201 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 19:38:08.157310 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.157236 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wxt6l" Jul 15 19:38:08.157310 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.157248 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 19:38:08.157417 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.157370 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 19:38:08.157767 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.157734 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r8pvn\"" Jul 15 19:38:08.158586 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.158565 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8vx9q\"" Jul 15 19:38:08.158837 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.158818 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 19:38:08.158837 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.158833 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.159014 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.158995 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 19:38:08.159482 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.159465 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:08.159588 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:08.159541 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv9rh" podUID="87560cf2-a9d6-46e4-86dd-eac6f20cc8c8" Jul 15 19:38:08.160592 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.160097 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 19:38:08.161785 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.160991 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mhtt6" Jul 15 19:38:08.161785 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.161147 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 19:38:08.161785 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.161200 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d8rdb\"" Jul 15 19:38:08.161785 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.161409 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 19:38:08.161785 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.161561 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 19:38:08.161785 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.161614 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x764r\"" Jul 15 19:38:08.161785 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.161633 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 19:38:08.162190 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.162176 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 19:38:08.162423 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.162408 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zsnk8" Jul 15 19:38:08.163946 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.162896 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 19:38:08.163946 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.162941 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 19:38:08.163946 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.162978 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 19:38:08.164217 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.164194 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.164370 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.164350 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 19:38:08.165025 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.165002 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 19:38:08.165542 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.165525 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 19:38:08.165831 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.165808 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 19:38:08.165922 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.165893 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 19:38:08.165922 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.165914 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:38:08.166248 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.166229 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m2vjk\"" Jul 15 19:38:08.166346 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.166330 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jj9x6\"" Jul 15 19:38:08.166405 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.166234 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.166953 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.166931 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k26sg\"" Jul 15 19:38:08.167311 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.167296 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 19:38:08.168046 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.168023 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:38:08.169763 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.169729 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.169952 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.169936 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-log-socket\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.170034 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.169959 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twc4m\" (UniqueName: \"kubernetes.io/projected/33e0c74e-e752-4c0f-9c26-a9742b866f41-kube-api-access-twc4m\") pod \"network-metrics-daemon-477vh\" (UID: \"33e0c74e-e752-4c0f-9c26-a9742b866f41\") " pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:08.170034 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.169982 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.170034 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170000 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbt9r\" (UniqueName: \"kubernetes.io/projected/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-kube-api-access-dbt9r\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.170034 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170028 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwhn9\" (UniqueName: \"kubernetes.io/projected/2d86b57f-2938-41d5-8d6e-d2a17177030d-kube-api-access-fwhn9\") pod \"node-ca-wxt6l\" (UID: \"2d86b57f-2938-41d5-8d6e-d2a17177030d\") " pod="openshift-image-registry/node-ca-wxt6l" Jul 15 19:38:08.170176 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170103 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 19:38:08.170176 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170142 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-cni-bin\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.170176 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170164 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-ovnkube-config\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.170261 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170180 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsvq9\" (UniqueName: \"kubernetes.io/projected/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-kube-api-access-qsvq9\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.170261 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170211 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67b2bcf5-c147-4a76-b620-b0cec94a0132-hosts-file\") pod \"node-resolver-zswzk\" (UID: \"67b2bcf5-c147-4a76-b620-b0cec94a0132\") " pod="openshift-dns/node-resolver-zswzk" Jul 15 19:38:08.170261 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170227 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67b2bcf5-c147-4a76-b620-b0cec94a0132-tmp-dir\") pod \"node-resolver-zswzk\" (UID: \"67b2bcf5-c147-4a76-b620-b0cec94a0132\") " pod="openshift-dns/node-resolver-zswzk" Jul 15 19:38:08.170261 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170244 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2d86b57f-2938-41d5-8d6e-d2a17177030d-serviceca\") pod \"node-ca-wxt6l\" (UID: \"2d86b57f-2938-41d5-8d6e-d2a17177030d\") " pod="openshift-image-registry/node-ca-wxt6l" Jul 15 19:38:08.170374 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170297 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-run-systemd\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.170374 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170314 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-etc-selinux\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.170374 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170328 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-var-lib-openvswitch\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.170374 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170350 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-run-ovn-kubernetes\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.170374 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170365 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-ovn-node-metrics-cert\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.170516 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170378 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-ovnkube-script-lib\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.170516 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170392 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-registration-dir\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.170516 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170420 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret\") pod \"global-pull-secret-syncer-rv9rh\" (UID: \"87560cf2-a9d6-46e4-86dd-eac6f20cc8c8\") " pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:08.170516 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170437 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-kubelet\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.170516 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170450 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-run-netns\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.170516 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170464 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-cni-netd\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.170516 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170480 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.170516 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170496 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-device-dir\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.170516 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170510 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-sys-fs\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.170769 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170546 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pqrc\" (UniqueName: \"kubernetes.io/projected/67b2bcf5-c147-4a76-b620-b0cec94a0132-kube-api-access-4pqrc\") pod \"node-resolver-zswzk\" (UID: \"67b2bcf5-c147-4a76-b620-b0cec94a0132\") " pod="openshift-dns/node-resolver-zswzk" Jul 15 19:38:08.170769 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170559 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d86b57f-2938-41d5-8d6e-d2a17177030d-host\") pod \"node-ca-wxt6l\" (UID: \"2d86b57f-2938-41d5-8d6e-d2a17177030d\") " pod="openshift-image-registry/node-ca-wxt6l" Jul 15 19:38:08.170769 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170571 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs\") pod \"network-metrics-daemon-477vh\" (UID: \"33e0c74e-e752-4c0f-9c26-a9742b866f41\") " pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:08.170769 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170585 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfzvq\" (UniqueName: \"kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq\") pod \"network-check-target-2ws6h\" (UID: \"9db8df3b-ca71-41fb-9039-a9b1afc9f1bd\") " pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:08.170769 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170598 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-node-log\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.170769 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170610 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-systemd-units\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.170769 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170651 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-slash\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.170769 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170680 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-etc-openvswitch\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.170769 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170710 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-run-ovn\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.170769 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170722 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 19:38:08.170769 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170728 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-env-overrides\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.171178 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170777 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-run-openvswitch\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.171178 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170802 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a81be65-7d76-41b7-962a-c0380b732b1d-agent-certs\") pod \"konnectivity-agent-mhtt6\" (UID: \"1a81be65-7d76-41b7-962a-c0380b732b1d\") " pod="kube-system/konnectivity-agent-mhtt6" Jul 15 19:38:08.171178 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170830 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a81be65-7d76-41b7-962a-c0380b732b1d-konnectivity-ca\") pod \"konnectivity-agent-mhtt6\" (UID: \"1a81be65-7d76-41b7-962a-c0380b732b1d\") " pod="kube-system/konnectivity-agent-mhtt6" Jul 15 19:38:08.171178 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170854 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-socket-dir\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.171178 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170883 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-kubelet-config\") pod \"global-pull-secret-syncer-rv9rh\" (UID: \"87560cf2-a9d6-46e4-86dd-eac6f20cc8c8\") " pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:08.171178 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170856 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 19:38:08.171178 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170928 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wvsqr\"" Jul 15 19:38:08.171178 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170937 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-dbus\") pod \"global-pull-secret-syncer-rv9rh\" (UID: \"87560cf2-a9d6-46e4-86dd-eac6f20cc8c8\") " pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:08.171178 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.170956 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 19:38:08.171178 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.171089 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 19:38:08.172252 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.172231 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 19:38:08.172252 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.172248 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-r7v2k\"" Jul 15 19:38:08.173426 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.173388 2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:38:08.174324 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.174301 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 19:38:08.206919 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.206897 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h9dfz" Jul 15 19:38:08.219735 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.219400 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h9dfz" Jul 15 19:38:08.261945 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.261919 2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 19:38:08.271944 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.271915 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-var-lib-openvswitch\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.272107 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.271956 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-ovn-node-metrics-cert\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.272107 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.271984 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-ovnkube-script-lib\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.272107 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272012 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-cni-binary-copy\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.272107 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272035 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-kubelet\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.272107 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272038 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-var-lib-openvswitch\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.272107 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272061 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-run-netns\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.272107 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272086 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-modprobe-d\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.272107 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272101 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-kubelet\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.272107 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272108 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-sysctl-d\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.272509 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272131 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-systemd\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.272509 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272147 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-run-netns\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.272509 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272154 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-sys-fs\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.272509 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272194 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d86b57f-2938-41d5-8d6e-d2a17177030d-host\") pod \"node-ca-wxt6l\" (UID: \"2d86b57f-2938-41d5-8d6e-d2a17177030d\") " pod="openshift-image-registry/node-ca-wxt6l" Jul 15 19:38:08.272509 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272248 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d86b57f-2938-41d5-8d6e-d2a17177030d-host\") pod \"node-ca-wxt6l\" (UID: \"2d86b57f-2938-41d5-8d6e-d2a17177030d\") " pod="openshift-image-registry/node-ca-wxt6l" Jul 15 19:38:08.272509 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272263 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-sys-fs\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.272509 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272270 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs\") pod \"network-metrics-daemon-477vh\" (UID: \"33e0c74e-e752-4c0f-9c26-a9742b866f41\") " pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:08.272509 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272312 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-systemd-units\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.272509 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272335 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-run-ovn\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.272509 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272361 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7kd6\" (UniqueName: \"kubernetes.io/projected/210f6641-2b87-4b63-8c2b-ccddf386feb7-kube-api-access-f7kd6\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.272509 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:08.272369 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:38:08.272509 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272387 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-host-run-k8s-cni-cncf-io\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.272509 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272409 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-host-run-netns\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.272509 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272450 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-run-ovn\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.272509 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272409 2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 19:38:08.272509 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:08.272497 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs podName:33e0c74e-e752-4c0f-9c26-a9742b866f41 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:08.772412139 +0000 UTC m=+3.016918593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs") pod "network-metrics-daemon-477vh" (UID: "33e0c74e-e752-4c0f-9c26-a9742b866f41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:38:08.273227 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272581 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-hostroot\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.273227 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272599 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c865faf5-fd82-4070-b169-a018a5c0bb0d-cnibin\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.273227 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272617 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a81be65-7d76-41b7-962a-c0380b732b1d-agent-certs\") pod \"konnectivity-agent-mhtt6\" (UID: \"1a81be65-7d76-41b7-962a-c0380b732b1d\") " pod="kube-system/konnectivity-agent-mhtt6" Jul 15 19:38:08.273227 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272631 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-socket-dir\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.273227 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272631 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-ovnkube-script-lib\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.273227 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272647 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-kubelet-config\") pod \"global-pull-secret-syncer-rv9rh\" (UID: \"87560cf2-a9d6-46e4-86dd-eac6f20cc8c8\") " pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:08.273227 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272683 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/706dc393-7eee-447b-bb31-6f52d2a27d70-iptables-alerter-script\") pod \"iptables-alerter-zsnk8\" (UID: \"706dc393-7eee-447b-bb31-6f52d2a27d70\") " pod="openshift-network-operator/iptables-alerter-zsnk8" Jul 15 19:38:08.273227 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272766 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-kubelet-config\") pod \"global-pull-secret-syncer-rv9rh\" (UID: \"87560cf2-a9d6-46e4-86dd-eac6f20cc8c8\") " pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:08.273227 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272815 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-var-lib-kubelet\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.273227 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272822 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-socket-dir\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.273227 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272373 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-systemd-units\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.273227 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272848 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-host-var-lib-kubelet\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.273227 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272871 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-log-socket\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.273227 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272894 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twc4m\" (UniqueName: \"kubernetes.io/projected/33e0c74e-e752-4c0f-9c26-a9742b866f41-kube-api-access-twc4m\") pod \"network-metrics-daemon-477vh\" (UID: \"33e0c74e-e752-4c0f-9c26-a9742b866f41\") " pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:08.273227 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272902 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-log-socket\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.273227 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272917 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.273227 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272939 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-sys\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.274035 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272964 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-etc-kubernetes\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.274035 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272987 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-cni-bin\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.274035 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.272987 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.274035 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273006 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-ovnkube-config\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.274035 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273027 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67b2bcf5-c147-4a76-b620-b0cec94a0132-tmp-dir\") pod \"node-resolver-zswzk\" (UID: \"67b2bcf5-c147-4a76-b620-b0cec94a0132\") " pod="openshift-dns/node-resolver-zswzk" Jul 15 19:38:08.274035 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273058 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2d86b57f-2938-41d5-8d6e-d2a17177030d-serviceca\") pod \"node-ca-wxt6l\" (UID: \"2d86b57f-2938-41d5-8d6e-d2a17177030d\") " pod="openshift-image-registry/node-ca-wxt6l" Jul 15 19:38:08.274035 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273087 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c865faf5-fd82-4070-b169-a018a5c0bb0d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.274035 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273117 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-run-systemd\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.274035 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273141 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-sysconfig\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.274035 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273166 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8msp6\" (UniqueName: \"kubernetes.io/projected/c865faf5-fd82-4070-b169-a018a5c0bb0d-kube-api-access-8msp6\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.274035 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273119 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-cni-bin\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.274035 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273192 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-run-ovn-kubernetes\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.274035 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273169 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-run-systemd\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.274035 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273240 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-run-ovn-kubernetes\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.274035 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273274 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-registration-dir\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.274035 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273276 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67b2bcf5-c147-4a76-b620-b0cec94a0132-tmp-dir\") pod \"node-resolver-zswzk\" (UID: \"67b2bcf5-c147-4a76-b620-b0cec94a0132\") " pod="openshift-dns/node-resolver-zswzk" Jul 15 19:38:08.274035 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273307 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret\") pod \"global-pull-secret-syncer-rv9rh\" (UID: \"87560cf2-a9d6-46e4-86dd-eac6f20cc8c8\") " pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:08.274035 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:08.273373 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:38:08.274882 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273368 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-registration-dir\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.274882 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273396 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-multus-conf-dir\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.274882 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273425 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2d86b57f-2938-41d5-8d6e-d2a17177030d-serviceca\") pod \"node-ca-wxt6l\" (UID: \"2d86b57f-2938-41d5-8d6e-d2a17177030d\") " pod="openshift-image-registry/node-ca-wxt6l" Jul 15 19:38:08.274882 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:08.273434 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret podName:87560cf2-a9d6-46e4-86dd-eac6f20cc8c8 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:08.77341973 +0000 UTC m=+3.017926176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret") pod "global-pull-secret-syncer-rv9rh" (UID: "87560cf2-a9d6-46e4-86dd-eac6f20cc8c8") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:38:08.274882 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273452 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-sysctl-conf\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.274882 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273476 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-system-cni-dir\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.274882 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273500 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-host-var-lib-cni-multus\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.274882 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273524 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c865faf5-fd82-4070-b169-a018a5c0bb0d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.274882 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273562 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-cni-netd\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.274882 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273635 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.274882 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273638 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-cni-netd\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.274882 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273679 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-device-dir\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.274882 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273730 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pqrc\" (UniqueName: \"kubernetes.io/projected/67b2bcf5-c147-4a76-b620-b0cec94a0132-kube-api-access-4pqrc\") pod \"node-resolver-zswzk\" (UID: \"67b2bcf5-c147-4a76-b620-b0cec94a0132\") " pod="openshift-dns/node-resolver-zswzk" Jul 15 19:38:08.274882 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273732 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-device-dir\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.274882 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273803 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.274882 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273859 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szhkt\" (UniqueName: \"kubernetes.io/projected/706dc393-7eee-447b-bb31-6f52d2a27d70-kube-api-access-szhkt\") pod \"iptables-alerter-zsnk8\" (UID: \"706dc393-7eee-447b-bb31-6f52d2a27d70\") " pod="openshift-network-operator/iptables-alerter-zsnk8" Jul 15 19:38:08.275621 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273882 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/210f6641-2b87-4b63-8c2b-ccddf386feb7-tmp\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.275621 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273905 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c865faf5-fd82-4070-b169-a018a5c0bb0d-system-cni-dir\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.275621 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273928 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/706dc393-7eee-447b-bb31-6f52d2a27d70-host-slash\") pod \"iptables-alerter-zsnk8\" (UID: \"706dc393-7eee-447b-bb31-6f52d2a27d70\") " pod="openshift-network-operator/iptables-alerter-zsnk8" Jul 15 19:38:08.275621 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273949 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-lib-modules\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.275621 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273972 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzvq\" (UniqueName: \"kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq\") pod \"network-check-target-2ws6h\" (UID: \"9db8df3b-ca71-41fb-9039-a9b1afc9f1bd\") " pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:08.275621 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.273993 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-node-log\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.275621 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274024 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-slash\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.275621 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274044 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-etc-openvswitch\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.275621 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274073 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-env-overrides\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.275621 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274080 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-node-log\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.275621 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274096 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a81be65-7d76-41b7-962a-c0380b732b1d-konnectivity-ca\") pod \"konnectivity-agent-mhtt6\" (UID: \"1a81be65-7d76-41b7-962a-c0380b732b1d\") " pod="kube-system/konnectivity-agent-mhtt6" Jul 15 19:38:08.275621 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274118 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-run-openvswitch\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.275621 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274141 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-dbus\") pod \"global-pull-secret-syncer-rv9rh\" (UID: \"87560cf2-a9d6-46e4-86dd-eac6f20cc8c8\") " pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:08.275621 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274159 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-kubernetes\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.275621 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274176 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-run\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.275621 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274194 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-host\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.275621 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274212 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-tuned\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.276398 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274238 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-multus-cni-dir\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.276398 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274259 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-os-release\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.276398 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274283 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-run-openvswitch\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.276398 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274284 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbt9r\" (UniqueName: \"kubernetes.io/projected/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-kube-api-access-dbt9r\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.276398 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274327 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwhn9\" (UniqueName: \"kubernetes.io/projected/2d86b57f-2938-41d5-8d6e-d2a17177030d-kube-api-access-fwhn9\") pod \"node-ca-wxt6l\" (UID: \"2d86b57f-2938-41d5-8d6e-d2a17177030d\") " pod="openshift-image-registry/node-ca-wxt6l" Jul 15 19:38:08.276398 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274358 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-multus-socket-dir-parent\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.276398 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274381 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-multus-daemon-config\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.276398 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274407 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-host-run-multus-certs\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.276398 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274432 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxchw\" (UniqueName: \"kubernetes.io/projected/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-kube-api-access-qxchw\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.276398 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274455 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c865faf5-fd82-4070-b169-a018a5c0bb0d-os-release\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.276398 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274482 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsvq9\" (UniqueName: \"kubernetes.io/projected/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-kube-api-access-qsvq9\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.276398 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274508 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67b2bcf5-c147-4a76-b620-b0cec94a0132-hosts-file\") pod \"node-resolver-zswzk\" (UID: \"67b2bcf5-c147-4a76-b620-b0cec94a0132\") " pod="openshift-dns/node-resolver-zswzk" Jul 15 19:38:08.276398 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274547 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-cnibin\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.276398 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274573 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-host-var-lib-cni-bin\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.276398 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274599 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c865faf5-fd82-4070-b169-a018a5c0bb0d-cni-binary-copy\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.276398 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274614 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-dbus\") pod \"global-pull-secret-syncer-rv9rh\" (UID: \"87560cf2-a9d6-46e4-86dd-eac6f20cc8c8\") " pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:08.276398 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274623 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c865faf5-fd82-4070-b169-a018a5c0bb0d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.277091 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274649 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-etc-selinux\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.277091 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.274827 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-etc-selinux\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.277091 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.275088 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-env-overrides\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.277091 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.275196 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67b2bcf5-c147-4a76-b620-b0cec94a0132-hosts-file\") pod \"node-resolver-zswzk\" (UID: \"67b2bcf5-c147-4a76-b620-b0cec94a0132\") " pod="openshift-dns/node-resolver-zswzk" Jul 15 19:38:08.277091 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.275243 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-host-slash\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.277091 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.275291 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-etc-openvswitch\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.277091 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.275339 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a81be65-7d76-41b7-962a-c0380b732b1d-konnectivity-ca\") pod \"konnectivity-agent-mhtt6\" (UID: \"1a81be65-7d76-41b7-962a-c0380b732b1d\") " pod="kube-system/konnectivity-agent-mhtt6" Jul 15 19:38:08.278029 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.277869 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a81be65-7d76-41b7-962a-c0380b732b1d-agent-certs\") pod \"konnectivity-agent-mhtt6\" (UID: \"1a81be65-7d76-41b7-962a-c0380b732b1d\") " pod="kube-system/konnectivity-agent-mhtt6" Jul 15 19:38:08.278130 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.278095 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-ovnkube-config\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.280132 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.280113 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-ovn-node-metrics-cert\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.283534 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:08.283518 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:38:08.283634 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:08.283542 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:38:08.283634 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:08.283555 2554 projected.go:196] Error preparing data for projected volume kube-api-access-jfzvq for pod openshift-network-diagnostics/network-check-target-2ws6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:38:08.283634 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:08.283615 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq podName:9db8df3b-ca71-41fb-9039-a9b1afc9f1bd nodeName:}" failed. No retries permitted until 2026-07-15 19:38:08.783598626 +0000 UTC m=+3.028105088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jfzvq" (UniqueName: "kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq") pod "network-check-target-2ws6h" (UID: "9db8df3b-ca71-41fb-9039-a9b1afc9f1bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:38:08.285311 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.285263 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twc4m\" (UniqueName: \"kubernetes.io/projected/33e0c74e-e752-4c0f-9c26-a9742b866f41-kube-api-access-twc4m\") pod \"network-metrics-daemon-477vh\" (UID: \"33e0c74e-e752-4c0f-9c26-a9742b866f41\") " pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:08.285601 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.285581 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pqrc\" (UniqueName: \"kubernetes.io/projected/67b2bcf5-c147-4a76-b620-b0cec94a0132-kube-api-access-4pqrc\") pod \"node-resolver-zswzk\" (UID: \"67b2bcf5-c147-4a76-b620-b0cec94a0132\") " pod="openshift-dns/node-resolver-zswzk" Jul 15 19:38:08.286448 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.286428 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbt9r\" (UniqueName: \"kubernetes.io/projected/49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1-kube-api-access-dbt9r\") pod \"aws-ebs-csi-driver-node-lswmx\" (UID: \"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.286638 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.286621 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwhn9\" (UniqueName: \"kubernetes.io/projected/2d86b57f-2938-41d5-8d6e-d2a17177030d-kube-api-access-fwhn9\") pod \"node-ca-wxt6l\" (UID: \"2d86b57f-2938-41d5-8d6e-d2a17177030d\") " pod="openshift-image-registry/node-ca-wxt6l" Jul 15 19:38:08.286729 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.286713 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsvq9\" (UniqueName: \"kubernetes.io/projected/16c0161e-1bfd-4b7b-93e0-a0da2acad94f-kube-api-access-qsvq9\") pod \"ovnkube-node-nxd2f\" (UID: \"16c0161e-1bfd-4b7b-93e0-a0da2acad94f\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.375373 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375335 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c865faf5-fd82-4070-b169-a018a5c0bb0d-cnibin\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.375567 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375386 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/706dc393-7eee-447b-bb31-6f52d2a27d70-iptables-alerter-script\") pod \"iptables-alerter-zsnk8\" (UID: \"706dc393-7eee-447b-bb31-6f52d2a27d70\") " pod="openshift-network-operator/iptables-alerter-zsnk8" Jul 15 19:38:08.375567 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375410 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-var-lib-kubelet\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.375567 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375429 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-host-var-lib-kubelet\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.375567 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375445 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-sys\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.375567 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375454 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c865faf5-fd82-4070-b169-a018a5c0bb0d-cnibin\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.375567 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375461 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-etc-kubernetes\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.375567 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375498 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-etc-kubernetes\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.375567 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375498 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c865faf5-fd82-4070-b169-a018a5c0bb0d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.375991 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375661 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-sys\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.375991 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375677 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c865faf5-fd82-4070-b169-a018a5c0bb0d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.375991 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375702 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-sysconfig\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.375991 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375716 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-host-var-lib-kubelet\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.375991 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375733 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8msp6\" (UniqueName: \"kubernetes.io/projected/c865faf5-fd82-4070-b169-a018a5c0bb0d-kube-api-access-8msp6\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.375991 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375785 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-sysconfig\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.375991 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375817 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-var-lib-kubelet\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.375991 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375842 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-multus-conf-dir\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.375991 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375872 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-sysctl-conf\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.375991 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375895 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-system-cni-dir\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.375991 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375918 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-host-var-lib-cni-multus\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.375991 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375944 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c865faf5-fd82-4070-b169-a018a5c0bb0d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.375991 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375958 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/706dc393-7eee-447b-bb31-6f52d2a27d70-iptables-alerter-script\") pod \"iptables-alerter-zsnk8\" (UID: \"706dc393-7eee-447b-bb31-6f52d2a27d70\") " pod="openshift-network-operator/iptables-alerter-zsnk8" Jul 15 19:38:08.375991 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.375975 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szhkt\" (UniqueName: \"kubernetes.io/projected/706dc393-7eee-447b-bb31-6f52d2a27d70-kube-api-access-szhkt\") pod \"iptables-alerter-zsnk8\" (UID: \"706dc393-7eee-447b-bb31-6f52d2a27d70\") " pod="openshift-network-operator/iptables-alerter-zsnk8" Jul 15 19:38:08.376629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376005 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-multus-conf-dir\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.376629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376005 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-system-cni-dir\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.376629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376035 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/210f6641-2b87-4b63-8c2b-ccddf386feb7-tmp\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.376629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376053 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c865faf5-fd82-4070-b169-a018a5c0bb0d-system-cni-dir\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.376629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376068 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/706dc393-7eee-447b-bb31-6f52d2a27d70-host-slash\") pod \"iptables-alerter-zsnk8\" (UID: \"706dc393-7eee-447b-bb31-6f52d2a27d70\") " pod="openshift-network-operator/iptables-alerter-zsnk8" Jul 15 19:38:08.376629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376085 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-lib-modules\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.376629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376089 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-sysctl-conf\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.376629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376116 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-kubernetes\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.376629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376118 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c865faf5-fd82-4070-b169-a018a5c0bb0d-system-cni-dir\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.376629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376131 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-run\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.376629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376146 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-host\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.376629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376160 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-tuned\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.376629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376176 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-multus-cni-dir\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.376629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376192 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-os-release\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.376629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376209 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-multus-socket-dir-parent\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.376629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376209 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-host-var-lib-cni-multus\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.376629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376223 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-multus-daemon-config\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.376629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376228 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-run\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.377419 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376239 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-host-run-multus-certs\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.377419 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376261 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-host-run-multus-certs\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.377419 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376264 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxchw\" (UniqueName: \"kubernetes.io/projected/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-kube-api-access-qxchw\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.377419 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376296 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c865faf5-fd82-4070-b169-a018a5c0bb0d-os-release\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.377419 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376315 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-cnibin\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.377419 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376331 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-host-var-lib-cni-bin\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.377419 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376346 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c865faf5-fd82-4070-b169-a018a5c0bb0d-cni-binary-copy\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.377419 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376364 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c865faf5-fd82-4070-b169-a018a5c0bb0d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.377419 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376384 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-cni-binary-copy\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.377419 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376401 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-modprobe-d\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.377419 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376415 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-sysctl-d\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.377419 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376424 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/706dc393-7eee-447b-bb31-6f52d2a27d70-host-slash\") pod \"iptables-alerter-zsnk8\" (UID: \"706dc393-7eee-447b-bb31-6f52d2a27d70\") " pod="openshift-network-operator/iptables-alerter-zsnk8" Jul 15 19:38:08.377419 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376431 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-systemd\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.377419 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376461 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7kd6\" (UniqueName: \"kubernetes.io/projected/210f6641-2b87-4b63-8c2b-ccddf386feb7-kube-api-access-f7kd6\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.377419 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376476 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-host-run-k8s-cni-cncf-io\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.377419 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376492 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-host-run-netns\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.377419 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376510 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-hostroot\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.377419 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376518 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-lib-modules\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.378081 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376564 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-host\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.378081 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376785 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c865faf5-fd82-4070-b169-a018a5c0bb0d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.378081 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376944 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c865faf5-fd82-4070-b169-a018a5c0bb0d-cni-binary-copy\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.378081 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.376995 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-hostroot\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.378081 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.377001 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-kubernetes\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.378081 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.377025 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-host-run-k8s-cni-cncf-io\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.378081 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.377033 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-os-release\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.378081 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.377090 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-multus-cni-dir\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.378081 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.377124 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-modprobe-d\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.378081 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.377132 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-sysctl-d\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.378081 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.377159 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-host-run-netns\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.378081 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.377096 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-multus-socket-dir-parent\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.378081 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.377213 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-host-var-lib-cni-bin\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.378081 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.377220 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c865faf5-fd82-4070-b169-a018a5c0bb0d-os-release\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.378081 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.377230 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-cnibin\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.378081 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.377181 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-systemd\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.378081 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.377465 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c865faf5-fd82-4070-b169-a018a5c0bb0d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.378686 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.378110 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-cni-binary-copy\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.378686 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.378521 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/210f6641-2b87-4b63-8c2b-ccddf386feb7-tmp\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.378686 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.378523 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-multus-daemon-config\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.378883 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.378861 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/210f6641-2b87-4b63-8c2b-ccddf386feb7-etc-tuned\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.384729 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.384702 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szhkt\" (UniqueName: \"kubernetes.io/projected/706dc393-7eee-447b-bb31-6f52d2a27d70-kube-api-access-szhkt\") pod \"iptables-alerter-zsnk8\" (UID: \"706dc393-7eee-447b-bb31-6f52d2a27d70\") " pod="openshift-network-operator/iptables-alerter-zsnk8" Jul 15 19:38:08.384729 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.384723 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7kd6\" (UniqueName: \"kubernetes.io/projected/210f6641-2b87-4b63-8c2b-ccddf386feb7-kube-api-access-f7kd6\") pod \"tuned-487tv\" (UID: \"210f6641-2b87-4b63-8c2b-ccddf386feb7\") " pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.384976 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.384954 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8msp6\" (UniqueName: \"kubernetes.io/projected/c865faf5-fd82-4070-b169-a018a5c0bb0d-kube-api-access-8msp6\") pod \"multus-additional-cni-plugins-9mp58\" (UID: \"c865faf5-fd82-4070-b169-a018a5c0bb0d\") " pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.386080 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.386062 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxchw\" (UniqueName: \"kubernetes.io/projected/e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a-kube-api-access-qxchw\") pod \"multus-n9dx5\" (UID: \"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a\") " pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.471764 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.471666 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" Jul 15 19:38:08.476542 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.476521 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zswzk" Jul 15 19:38:08.485924 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.485893 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wxt6l" Jul 15 19:38:08.491550 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.491522 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:08.498198 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.498177 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mhtt6" Jul 15 19:38:08.505715 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.505695 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zsnk8" Jul 15 19:38:08.513264 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.513242 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-487tv" Jul 15 19:38:08.519839 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.519821 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9mp58" Jul 15 19:38:08.526400 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.526381 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n9dx5" Jul 15 19:38:08.779432 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.779343 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs\") pod \"network-metrics-daemon-477vh\" (UID: \"33e0c74e-e752-4c0f-9c26-a9742b866f41\") " pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:08.779432 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.779415 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret\") pod \"global-pull-secret-syncer-rv9rh\" (UID: \"87560cf2-a9d6-46e4-86dd-eac6f20cc8c8\") " pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:08.779655 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:08.779524 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:38:08.779655 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:08.779543 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:38:08.779655 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:08.779595 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret podName:87560cf2-a9d6-46e4-86dd-eac6f20cc8c8 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:09.779579262 +0000 UTC m=+4.024085701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret") pod "global-pull-secret-syncer-rv9rh" (UID: "87560cf2-a9d6-46e4-86dd-eac6f20cc8c8") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:38:08.779655 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:08.779615 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs podName:33e0c74e-e752-4c0f-9c26-a9742b866f41 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:09.779604766 +0000 UTC m=+4.024111206 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs") pod "network-metrics-daemon-477vh" (UID: "33e0c74e-e752-4c0f-9c26-a9742b866f41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:38:08.880130 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:08.880092 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzvq\" (UniqueName: \"kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq\") pod \"network-check-target-2ws6h\" (UID: \"9db8df3b-ca71-41fb-9039-a9b1afc9f1bd\") " pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:08.880318 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:08.880300 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:38:08.880370 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:08.880321 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:38:08.880370 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:08.880333 2554 projected.go:196] Error preparing data for projected volume kube-api-access-jfzvq for pod openshift-network-diagnostics/network-check-target-2ws6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:38:08.880444 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:08.880381 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq podName:9db8df3b-ca71-41fb-9039-a9b1afc9f1bd nodeName:}" failed. No retries permitted until 2026-07-15 19:38:09.880366503 +0000 UTC m=+4.124872942 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfzvq" (UniqueName: "kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq") pod "network-check-target-2ws6h" (UID: "9db8df3b-ca71-41fb-9039-a9b1afc9f1bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:38:09.020679 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:09.020634 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b2bcf5_c147_4a76_b620_b0cec94a0132.slice/crio-495d17cfa135779ac5f5e217edb0e6b542e6f8d5e53cb036cc6ad43ba35aaa62 WatchSource:0}: Error finding container 495d17cfa135779ac5f5e217edb0e6b542e6f8d5e53cb036cc6ad43ba35aaa62: Status 404 returned error can't find the container with id 495d17cfa135779ac5f5e217edb0e6b542e6f8d5e53cb036cc6ad43ba35aaa62 Jul 15 19:38:09.021608 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:09.021582 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ddad9c_733e_47dc_9edd_a7ef5ff1f76a.slice/crio-0c3c97a0bdb2f5bd2fe92dd281dd40c7798b0007b4a84c36fa2b1a168c95371d WatchSource:0}: Error finding container 0c3c97a0bdb2f5bd2fe92dd281dd40c7798b0007b4a84c36fa2b1a168c95371d: Status 404 returned error can't find the container with id 0c3c97a0bdb2f5bd2fe92dd281dd40c7798b0007b4a84c36fa2b1a168c95371d Jul 15 19:38:09.024299 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:09.024278 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210f6641_2b87_4b63_8c2b_ccddf386feb7.slice/crio-f579875347a2f5e65e1373daa063cb0c7b9133608f75047b7252338b83b830a8 WatchSource:0}: Error finding container f579875347a2f5e65e1373daa063cb0c7b9133608f75047b7252338b83b830a8: Status 404 returned error can't find the container with id f579875347a2f5e65e1373daa063cb0c7b9133608f75047b7252338b83b830a8 Jul 15 19:38:09.026214 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:09.026161 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d86b57f_2938_41d5_8d6e_d2a17177030d.slice/crio-6e1e6cbb899c2b5108aeabdce7f090e01634a0b8036ea5ab077c4a156889283f WatchSource:0}: Error finding container 6e1e6cbb899c2b5108aeabdce7f090e01634a0b8036ea5ab077c4a156889283f: Status 404 returned error can't find the container with id 6e1e6cbb899c2b5108aeabdce7f090e01634a0b8036ea5ab077c4a156889283f Jul 15 19:38:09.032571 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:09.030116 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c0161e_1bfd_4b7b_93e0_a0da2acad94f.slice/crio-8afcb8dae3e47f223b942a1a5438f4c0f4a293257eef4a40ee7416361207d741 WatchSource:0}: Error finding container 8afcb8dae3e47f223b942a1a5438f4c0f4a293257eef4a40ee7416361207d741: Status 404 returned error can't find the container with id 8afcb8dae3e47f223b942a1a5438f4c0f4a293257eef4a40ee7416361207d741 Jul 15 19:38:09.032571 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:09.031020 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706dc393_7eee_447b_bb31_6f52d2a27d70.slice/crio-34e045a92feb1edf58c4b2a65cc33daa68df21c88cc0acc5d9448da7cc01ce50 WatchSource:0}: Error finding container 34e045a92feb1edf58c4b2a65cc33daa68df21c88cc0acc5d9448da7cc01ce50: Status 404 returned error can't find the container with id 34e045a92feb1edf58c4b2a65cc33daa68df21c88cc0acc5d9448da7cc01ce50 Jul 15 19:38:09.033721 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:09.033616 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a81be65_7d76_41b7_962a_c0380b732b1d.slice/crio-3a8184c24fb7bd651615c6628e83b6d5d36da6e4629e06d64d1e40e9c0df396b WatchSource:0}: Error finding container 3a8184c24fb7bd651615c6628e83b6d5d36da6e4629e06d64d1e40e9c0df396b: Status 404 returned error can't find the container with id 3a8184c24fb7bd651615c6628e83b6d5d36da6e4629e06d64d1e40e9c0df396b Jul 15 19:38:09.222582 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:09.222341 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 19:33:08 +0000 UTC" deadline="2028-05-02 11:25:43.761284422 +0000 UTC" Jul 15 19:38:09.222582 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:09.222581 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15759h47m34.538708573s" Jul 15 19:38:09.232833 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:09.232810 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:09.232942 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:09.232922 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-477vh" podUID="33e0c74e-e752-4c0f-9c26-a9742b866f41" Jul 15 19:38:09.241686 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:09.241653 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-139.ec2.internal" event={"ID":"4dd0763a6110d8ba2bd7fcafeff6d0d7","Type":"ContainerStarted","Data":"329fc7666db7116d4f978a1abbad6fb4abcd8f164aee99370970644401d7ed2d"} Jul 15 19:38:09.242931 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:09.242905 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zsnk8" event={"ID":"706dc393-7eee-447b-bb31-6f52d2a27d70","Type":"ContainerStarted","Data":"34e045a92feb1edf58c4b2a65cc33daa68df21c88cc0acc5d9448da7cc01ce50"} Jul 15 19:38:09.244019 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:09.243991 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" event={"ID":"16c0161e-1bfd-4b7b-93e0-a0da2acad94f","Type":"ContainerStarted","Data":"8afcb8dae3e47f223b942a1a5438f4c0f4a293257eef4a40ee7416361207d741"} Jul 15 19:38:09.245100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:09.245065 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" event={"ID":"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1","Type":"ContainerStarted","Data":"e12c0efbb140799dc4a3f22a5f1d316cd48ee1be1575c405407caa5e68235e3c"} Jul 15 19:38:09.246065 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:09.246038 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wxt6l" event={"ID":"2d86b57f-2938-41d5-8d6e-d2a17177030d","Type":"ContainerStarted","Data":"6e1e6cbb899c2b5108aeabdce7f090e01634a0b8036ea5ab077c4a156889283f"} Jul 15 19:38:09.247216 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:09.247195 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n9dx5" event={"ID":"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a","Type":"ContainerStarted","Data":"0c3c97a0bdb2f5bd2fe92dd281dd40c7798b0007b4a84c36fa2b1a168c95371d"} Jul 15 19:38:09.248250 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:09.248230 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zswzk" event={"ID":"67b2bcf5-c147-4a76-b620-b0cec94a0132","Type":"ContainerStarted","Data":"495d17cfa135779ac5f5e217edb0e6b542e6f8d5e53cb036cc6ad43ba35aaa62"} Jul 15 19:38:09.249351 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:09.249327 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9mp58" event={"ID":"c865faf5-fd82-4070-b169-a018a5c0bb0d","Type":"ContainerStarted","Data":"24f662aee7a6bfdbea09bc1d04796a87ead8383183ac1122058bf6bb51c8e10e"} Jul 15 19:38:09.250391 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:09.250369 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mhtt6" event={"ID":"1a81be65-7d76-41b7-962a-c0380b732b1d","Type":"ContainerStarted","Data":"3a8184c24fb7bd651615c6628e83b6d5d36da6e4629e06d64d1e40e9c0df396b"} Jul 15 19:38:09.251414 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:09.251396 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-487tv" event={"ID":"210f6641-2b87-4b63-8c2b-ccddf386feb7","Type":"ContainerStarted","Data":"f579875347a2f5e65e1373daa063cb0c7b9133608f75047b7252338b83b830a8"} Jul 15 19:38:09.787195 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:09.787159 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs\") pod \"network-metrics-daemon-477vh\" (UID: \"33e0c74e-e752-4c0f-9c26-a9742b866f41\") " pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:09.787360 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:09.787227 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret\") pod \"global-pull-secret-syncer-rv9rh\" (UID: \"87560cf2-a9d6-46e4-86dd-eac6f20cc8c8\") " pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:09.787419 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:09.787406 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:38:09.787484 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:09.787470 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret podName:87560cf2-a9d6-46e4-86dd-eac6f20cc8c8 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:11.787452786 +0000 UTC m=+6.031959241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret") pod "global-pull-secret-syncer-rv9rh" (UID: "87560cf2-a9d6-46e4-86dd-eac6f20cc8c8") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:38:09.787886 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:09.787868 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:38:09.787961 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:09.787933 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs podName:33e0c74e-e752-4c0f-9c26-a9742b866f41 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:11.787914277 +0000 UTC m=+6.032420728 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs") pod "network-metrics-daemon-477vh" (UID: "33e0c74e-e752-4c0f-9c26-a9742b866f41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:38:09.888610 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:09.888005 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzvq\" (UniqueName: \"kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq\") pod \"network-check-target-2ws6h\" (UID: \"9db8df3b-ca71-41fb-9039-a9b1afc9f1bd\") " pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:09.888610 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:09.888181 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:38:09.888610 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:09.888198 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:38:09.888610 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:09.888210 2554 projected.go:196] Error preparing data for projected volume kube-api-access-jfzvq for pod openshift-network-diagnostics/network-check-target-2ws6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:38:09.888610 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:09.888267 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq podName:9db8df3b-ca71-41fb-9039-a9b1afc9f1bd nodeName:}" failed. No retries permitted until 2026-07-15 19:38:11.888249639 +0000 UTC m=+6.132756081 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfzvq" (UniqueName: "kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq") pod "network-check-target-2ws6h" (UID: "9db8df3b-ca71-41fb-9039-a9b1afc9f1bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:38:10.222844 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:10.222735 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 19:33:08 +0000 UTC" deadline="2028-04-12 15:53:28.046748944 +0000 UTC" Jul 15 19:38:10.222844 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:10.222794 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15284h15m17.823960455s" Jul 15 19:38:10.234736 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:10.234708 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:10.234929 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:10.234853 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv9rh" podUID="87560cf2-a9d6-46e4-86dd-eac6f20cc8c8" Jul 15 19:38:10.235264 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:10.235245 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:10.235371 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:10.235349 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ws6h" podUID="9db8df3b-ca71-41fb-9039-a9b1afc9f1bd" Jul 15 19:38:10.265711 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:10.264743 2554 generic.go:356] "Generic (PLEG): container finished" podID="ff7075f5bca7a3a4c83421a497bc75eb" containerID="c7c838ff5be7ff23218559be9bbade2ab841dbfe15d5b92ced8f1a6f951f52fd" exitCode=0 Jul 15 19:38:10.265711 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:10.265367 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal" event={"ID":"ff7075f5bca7a3a4c83421a497bc75eb","Type":"ContainerDied","Data":"c7c838ff5be7ff23218559be9bbade2ab841dbfe15d5b92ced8f1a6f951f52fd"} Jul 15 19:38:10.283142 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:10.283086 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-139.ec2.internal" podStartSLOduration=3.283054447 podStartE2EDuration="3.283054447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:09.258875992 +0000 UTC m=+3.503382450" watchObservedRunningTime="2026-07-15 19:38:10.283054447 +0000 UTC m=+4.527560911" Jul 15 19:38:11.232921 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:11.232871 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:11.233406 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:11.233016 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-477vh" podUID="33e0c74e-e752-4c0f-9c26-a9742b866f41" Jul 15 19:38:11.294330 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:11.294285 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal" event={"ID":"ff7075f5bca7a3a4c83421a497bc75eb","Type":"ContainerStarted","Data":"b4c499067b2049df5eb67cbab7b0230f9d816e1eb65f2ae6d90590fc6a6b6c9c"} Jul 15 19:38:11.805240 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:11.804419 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret\") pod \"global-pull-secret-syncer-rv9rh\" (UID: \"87560cf2-a9d6-46e4-86dd-eac6f20cc8c8\") " pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:11.805240 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:11.804515 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs\") pod \"network-metrics-daemon-477vh\" (UID: \"33e0c74e-e752-4c0f-9c26-a9742b866f41\") " pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:11.805240 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:11.804660 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:38:11.805240 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:11.804732 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs podName:33e0c74e-e752-4c0f-9c26-a9742b866f41 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:15.804704484 +0000 UTC m=+10.049210933 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs") pod "network-metrics-daemon-477vh" (UID: "33e0c74e-e752-4c0f-9c26-a9742b866f41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:38:11.805240 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:11.805159 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:38:11.805240 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:11.805207 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret podName:87560cf2-a9d6-46e4-86dd-eac6f20cc8c8 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:15.80519191 +0000 UTC m=+10.049698349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret") pod "global-pull-secret-syncer-rv9rh" (UID: "87560cf2-a9d6-46e4-86dd-eac6f20cc8c8") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:38:11.905727 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:11.905687 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzvq\" (UniqueName: \"kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq\") pod \"network-check-target-2ws6h\" (UID: \"9db8df3b-ca71-41fb-9039-a9b1afc9f1bd\") " pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:11.905935 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:11.905915 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:38:11.905987 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:11.905942 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:38:11.905987 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:11.905955 2554 projected.go:196] Error preparing data for projected volume kube-api-access-jfzvq for pod openshift-network-diagnostics/network-check-target-2ws6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:38:11.906054 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:11.906015 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq podName:9db8df3b-ca71-41fb-9039-a9b1afc9f1bd nodeName:}" failed. No retries permitted until 2026-07-15 19:38:15.905996523 +0000 UTC m=+10.150502968 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfzvq" (UniqueName: "kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq") pod "network-check-target-2ws6h" (UID: "9db8df3b-ca71-41fb-9039-a9b1afc9f1bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:38:12.233983 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:12.233904 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:12.233983 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:12.233945 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:12.234488 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:12.234031 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ws6h" podUID="9db8df3b-ca71-41fb-9039-a9b1afc9f1bd" Jul 15 19:38:12.234488 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:12.234204 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv9rh" podUID="87560cf2-a9d6-46e4-86dd-eac6f20cc8c8" Jul 15 19:38:13.232334 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:13.232299 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:13.232519 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:13.232459 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-477vh" podUID="33e0c74e-e752-4c0f-9c26-a9742b866f41" Jul 15 19:38:14.233852 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:14.233146 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:14.233852 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:14.233275 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ws6h" podUID="9db8df3b-ca71-41fb-9039-a9b1afc9f1bd" Jul 15 19:38:14.233852 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:14.233672 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:14.233852 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:14.233789 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv9rh" podUID="87560cf2-a9d6-46e4-86dd-eac6f20cc8c8" Jul 15 19:38:15.233241 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:15.233157 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:15.233409 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:15.233301 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-477vh" podUID="33e0c74e-e752-4c0f-9c26-a9742b866f41" Jul 15 19:38:15.845608 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:15.845569 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret\") pod \"global-pull-secret-syncer-rv9rh\" (UID: \"87560cf2-a9d6-46e4-86dd-eac6f20cc8c8\") " pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:15.846111 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:15.845649 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs\") pod \"network-metrics-daemon-477vh\" (UID: \"33e0c74e-e752-4c0f-9c26-a9742b866f41\") " pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:15.846111 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:15.845703 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:38:15.846111 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:15.845738 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:38:15.846111 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:15.845801 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs podName:33e0c74e-e752-4c0f-9c26-a9742b866f41 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:23.845783449 +0000 UTC m=+18.090289912 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs") pod "network-metrics-daemon-477vh" (UID: "33e0c74e-e752-4c0f-9c26-a9742b866f41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:38:15.846111 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:15.845815 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret podName:87560cf2-a9d6-46e4-86dd-eac6f20cc8c8 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:23.845809161 +0000 UTC m=+18.090315600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret") pod "global-pull-secret-syncer-rv9rh" (UID: "87560cf2-a9d6-46e4-86dd-eac6f20cc8c8") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:38:15.946340 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:15.946302 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzvq\" (UniqueName: \"kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq\") pod \"network-check-target-2ws6h\" (UID: \"9db8df3b-ca71-41fb-9039-a9b1afc9f1bd\") " pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:15.946506 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:15.946471 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:38:15.946506 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:15.946497 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:38:15.946599 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:15.946509 2554 projected.go:196] Error preparing data for projected volume kube-api-access-jfzvq for pod openshift-network-diagnostics/network-check-target-2ws6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:38:15.946599 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:15.946578 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq podName:9db8df3b-ca71-41fb-9039-a9b1afc9f1bd nodeName:}" failed. No retries permitted until 2026-07-15 19:38:23.946559299 +0000 UTC m=+18.191065750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfzvq" (UniqueName: "kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq") pod "network-check-target-2ws6h" (UID: "9db8df3b-ca71-41fb-9039-a9b1afc9f1bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:38:16.238860 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:16.238743 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:16.239064 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:16.238996 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ws6h" podUID="9db8df3b-ca71-41fb-9039-a9b1afc9f1bd" Jul 15 19:38:16.239480 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:16.239451 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:16.239786 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:16.239724 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv9rh" podUID="87560cf2-a9d6-46e4-86dd-eac6f20cc8c8" Jul 15 19:38:17.233576 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:17.233081 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:17.233576 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:17.233225 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-477vh" podUID="33e0c74e-e752-4c0f-9c26-a9742b866f41" Jul 15 19:38:18.233806 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:18.233331 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:18.233806 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:18.233360 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:18.233806 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:18.233462 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv9rh" podUID="87560cf2-a9d6-46e4-86dd-eac6f20cc8c8" Jul 15 19:38:18.233806 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:18.233609 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ws6h" podUID="9db8df3b-ca71-41fb-9039-a9b1afc9f1bd" Jul 15 19:38:19.232524 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:19.232493 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:19.232689 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:19.232606 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-477vh" podUID="33e0c74e-e752-4c0f-9c26-a9742b866f41" Jul 15 19:38:20.232414 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:20.232374 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:20.232818 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:20.232374 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:20.232818 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:20.232528 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv9rh" podUID="87560cf2-a9d6-46e4-86dd-eac6f20cc8c8" Jul 15 19:38:20.232818 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:20.232584 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ws6h" podUID="9db8df3b-ca71-41fb-9039-a9b1afc9f1bd" Jul 15 19:38:21.233325 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:21.233283 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:21.233725 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:21.233413 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-477vh" podUID="33e0c74e-e752-4c0f-9c26-a9742b866f41" Jul 15 19:38:22.232545 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:22.232505 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:22.232684 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:22.232505 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:22.232684 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:22.232620 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ws6h" podUID="9db8df3b-ca71-41fb-9039-a9b1afc9f1bd" Jul 15 19:38:22.232786 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:22.232709 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv9rh" podUID="87560cf2-a9d6-46e4-86dd-eac6f20cc8c8" Jul 15 19:38:23.232880 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:23.232838 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:23.233474 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:23.232993 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-477vh" podUID="33e0c74e-e752-4c0f-9c26-a9742b866f41" Jul 15 19:38:23.913080 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:23.913046 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs\") pod \"network-metrics-daemon-477vh\" (UID: \"33e0c74e-e752-4c0f-9c26-a9742b866f41\") " pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:23.913290 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:23.913093 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret\") pod \"global-pull-secret-syncer-rv9rh\" (UID: \"87560cf2-a9d6-46e4-86dd-eac6f20cc8c8\") " pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:23.913290 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:23.913209 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:38:23.913290 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:23.913211 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:38:23.913290 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:23.913272 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret podName:87560cf2-a9d6-46e4-86dd-eac6f20cc8c8 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:39.913254818 +0000 UTC m=+34.157761258 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret") pod "global-pull-secret-syncer-rv9rh" (UID: "87560cf2-a9d6-46e4-86dd-eac6f20cc8c8") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:38:23.913290 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:23.913286 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs podName:33e0c74e-e752-4c0f-9c26-a9742b866f41 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:39.913279777 +0000 UTC m=+34.157786216 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs") pod "network-metrics-daemon-477vh" (UID: "33e0c74e-e752-4c0f-9c26-a9742b866f41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:38:24.013437 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:24.013397 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzvq\" (UniqueName: \"kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq\") pod \"network-check-target-2ws6h\" (UID: \"9db8df3b-ca71-41fb-9039-a9b1afc9f1bd\") " pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:24.013597 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:24.013575 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:38:24.013673 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:24.013598 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:38:24.013673 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:24.013609 2554 projected.go:196] Error preparing data for projected volume kube-api-access-jfzvq for pod openshift-network-diagnostics/network-check-target-2ws6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:38:24.013673 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:24.013657 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq podName:9db8df3b-ca71-41fb-9039-a9b1afc9f1bd nodeName:}" failed. No retries permitted until 2026-07-15 19:38:40.013644639 +0000 UTC m=+34.258151079 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfzvq" (UniqueName: "kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq") pod "network-check-target-2ws6h" (UID: "9db8df3b-ca71-41fb-9039-a9b1afc9f1bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:38:24.232930 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:24.232889 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:24.233382 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:24.233030 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ws6h" podUID="9db8df3b-ca71-41fb-9039-a9b1afc9f1bd" Jul 15 19:38:24.233382 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:24.233071 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:24.233382 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:24.233140 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv9rh" podUID="87560cf2-a9d6-46e4-86dd-eac6f20cc8c8" Jul 15 19:38:25.233114 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:25.233081 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:25.233563 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:25.233215 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-477vh" podUID="33e0c74e-e752-4c0f-9c26-a9742b866f41" Jul 15 19:38:26.233335 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:26.233295 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:26.233724 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:26.233399 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ws6h" podUID="9db8df3b-ca71-41fb-9039-a9b1afc9f1bd" Jul 15 19:38:26.233724 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:26.233523 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:26.233724 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:26.233614 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv9rh" podUID="87560cf2-a9d6-46e4-86dd-eac6f20cc8c8" Jul 15 19:38:27.233033 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:27.232997 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:27.233209 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:27.233133 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-477vh" podUID="33e0c74e-e752-4c0f-9c26-a9742b866f41" Jul 15 19:38:28.232827 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:28.232786 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:28.233271 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:28.232808 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:28.233271 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:28.232924 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ws6h" podUID="9db8df3b-ca71-41fb-9039-a9b1afc9f1bd" Jul 15 19:38:28.233271 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:28.232998 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv9rh" podUID="87560cf2-a9d6-46e4-86dd-eac6f20cc8c8" Jul 15 19:38:29.233194 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.232905 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:29.233890 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:29.233308 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-477vh" podUID="33e0c74e-e752-4c0f-9c26-a9742b866f41" Jul 15 19:38:29.334048 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.334024 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 19:38:29.334426 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.334403 2554 generic.go:356] "Generic (PLEG): container finished" podID="16c0161e-1bfd-4b7b-93e0-a0da2acad94f" containerID="5a684503d266404317cff2ac74dddffa986c6a21eb1862249eb7ab73b534541f" exitCode=1 Jul 15 19:38:29.334512 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.334482 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" event={"ID":"16c0161e-1bfd-4b7b-93e0-a0da2acad94f","Type":"ContainerStarted","Data":"aa619fef271b82eacaed088e0e4d097c6148a6c7316475b7f3f873a3e8b12a10"} Jul 15 19:38:29.334588 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.334571 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" event={"ID":"16c0161e-1bfd-4b7b-93e0-a0da2acad94f","Type":"ContainerStarted","Data":"dae682a652f222ce9af21b78b7f5e8eae4e587c7f9355596fa53f614f4c9f171"} Jul 15 19:38:29.334638 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.334627 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" event={"ID":"16c0161e-1bfd-4b7b-93e0-a0da2acad94f","Type":"ContainerStarted","Data":"5d3c26765b58106080fb0e9c2e3cf82d5a90e5732bbb1ade95c4bc0a001cf30d"} Jul 15 19:38:29.334685 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.334673 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" event={"ID":"16c0161e-1bfd-4b7b-93e0-a0da2acad94f","Type":"ContainerStarted","Data":"bfdd6d6d4b0eea9fb26623b0893977eb48d0235ad437fcec8d35e385280eb906"} Jul 15 19:38:29.334728 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.334715 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" event={"ID":"16c0161e-1bfd-4b7b-93e0-a0da2acad94f","Type":"ContainerDied","Data":"5a684503d266404317cff2ac74dddffa986c6a21eb1862249eb7ab73b534541f"} Jul 15 19:38:29.334799 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.334780 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" event={"ID":"16c0161e-1bfd-4b7b-93e0-a0da2acad94f","Type":"ContainerStarted","Data":"ed5a20b1d96540ad80b82376d9120d8422430aa5e673d9f43d1e92993d94bfe2"} Jul 15 19:38:29.335980 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.335959 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" event={"ID":"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1","Type":"ContainerStarted","Data":"2c1537988f50730f8e4153a8adc27ae3e2b71dfacb1167ebf00b5c1167c65365"} Jul 15 19:38:29.337196 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.337176 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wxt6l" event={"ID":"2d86b57f-2938-41d5-8d6e-d2a17177030d","Type":"ContainerStarted","Data":"0015dd98aee636df17f8f2a280f1e60386f6a09f0553b94c5b0b7237c1824cfe"} Jul 15 19:38:29.338638 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.338621 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n9dx5" event={"ID":"e1ddad9c-733e-47dc-9edd-a7ef5ff1f76a","Type":"ContainerStarted","Data":"85983f8d3ec1c8f61cdb892fe35737d1a196883655ec9737f0b6e8eda20ad162"} Jul 15 19:38:29.339950 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.339934 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zswzk" event={"ID":"67b2bcf5-c147-4a76-b620-b0cec94a0132","Type":"ContainerStarted","Data":"593ec3b50a46a0fd7492a5220ba6116ef8e4d02f30a18d983443108010c702cc"} Jul 15 19:38:29.341388 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.341357 2554 generic.go:356] "Generic (PLEG): container finished" podID="c865faf5-fd82-4070-b169-a018a5c0bb0d" containerID="7d1cdf635c2de9cf3cf666761020dc1e589d53e7236d76e3d36cd69a4f1de1cd" exitCode=0 Jul 15 19:38:29.341626 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.341435 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9mp58" event={"ID":"c865faf5-fd82-4070-b169-a018a5c0bb0d","Type":"ContainerDied","Data":"7d1cdf635c2de9cf3cf666761020dc1e589d53e7236d76e3d36cd69a4f1de1cd"} Jul 15 19:38:29.342957 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.342929 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mhtt6" event={"ID":"1a81be65-7d76-41b7-962a-c0380b732b1d","Type":"ContainerStarted","Data":"0bfcabf274e15690b308b2c30896a05104fb2f82b998ed60b2c5dbdd42c33b28"} Jul 15 19:38:29.344476 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.344458 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-487tv" event={"ID":"210f6641-2b87-4b63-8c2b-ccddf386feb7","Type":"ContainerStarted","Data":"d22ac4f555ed9467c84a2caed21e78f28af7de74d634eaf3788b3191049e31c1"} Jul 15 19:38:29.361018 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.360976 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-139.ec2.internal" podStartSLOduration=22.360963787 podStartE2EDuration="22.360963787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:11.313247323 +0000 UTC m=+5.557753786" watchObservedRunningTime="2026-07-15 19:38:29.360963787 +0000 UTC m=+23.605470248" Jul 15 19:38:29.386938 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.386884 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mhtt6" podStartSLOduration=3.905933093 podStartE2EDuration="23.386867793s" totalImagesPullingTime="19.4809347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:29.386679806 +0000 UTC m=+23.631186268" watchObservedRunningTime="2026-07-15 19:38:29.386867793 +0000 UTC m=+23.631374255" Jul 15 19:38:29.387143 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.387118 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wxt6l" podStartSLOduration=8.326171219 podStartE2EDuration="23.387107766s" totalImagesPullingTime="15.060936547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:29.36108505 +0000 UTC m=+23.605591493" watchObservedRunningTime="2026-07-15 19:38:29.387107766 +0000 UTC m=+23.631614226" Jul 15 19:38:29.409686 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.409633 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-487tv" podStartSLOduration=3.905029364 podStartE2EDuration="23.40961635s" totalImagesPullingTime="19.504586986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:29.40955606 +0000 UTC m=+23.654062521" watchObservedRunningTime="2026-07-15 19:38:29.40961635 +0000 UTC m=+23.654122811" Jul 15 19:38:29.476891 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.476842 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n9dx5" podStartSLOduration=3.900119683 podStartE2EDuration="23.47682759s" totalImagesPullingTime="19.576707907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:29.476500396 +0000 UTC m=+23.721006856" watchObservedRunningTime="2026-07-15 19:38:29.47682759 +0000 UTC m=+23.721334050" Jul 15 19:38:29.499889 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:29.499842 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zswzk" podStartSLOduration=4.009052549 podStartE2EDuration="23.499827301s" totalImagesPullingTime="19.490774752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:29.499566238 +0000 UTC m=+23.744072700" watchObservedRunningTime="2026-07-15 19:38:29.499827301 +0000 UTC m=+23.744333762" Jul 15 19:38:30.232839 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:30.232807 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:30.233036 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:30.232811 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:30.233036 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:30.232937 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ws6h" podUID="9db8df3b-ca71-41fb-9039-a9b1afc9f1bd" Jul 15 19:38:30.233036 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:30.233011 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv9rh" podUID="87560cf2-a9d6-46e4-86dd-eac6f20cc8c8" Jul 15 19:38:30.351223 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:30.351189 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zsnk8" event={"ID":"706dc393-7eee-447b-bb31-6f52d2a27d70","Type":"ContainerStarted","Data":"cca77ffe036a339ecb7e2161c8f6338bf1fa38e40bae3441b0bd0254676b3e41"} Jul 15 19:38:30.386406 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:30.386356 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zsnk8" podStartSLOduration=4.90538921 podStartE2EDuration="24.386333526s" totalImagesPullingTime="19.480944316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:30.385608389 +0000 UTC m=+24.630114844" watchObservedRunningTime="2026-07-15 19:38:30.386333526 +0000 UTC m=+24.630839989" Jul 15 19:38:30.503364 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:30.503337 2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 19:38:30.565386 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:30.565351 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mhtt6" Jul 15 19:38:30.565984 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:30.565962 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mhtt6" Jul 15 19:38:31.233294 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:31.233258 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:31.233571 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:31.233392 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-477vh" podUID="33e0c74e-e752-4c0f-9c26-a9742b866f41" Jul 15 19:38:31.242334 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:31.242234 2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T19:38:30.503358914Z","UUID":"2a53346d-6d93-4dde-bcbe-8e407abbb642","Handler":null,"Name":"","Endpoint":""} Jul 15 19:38:31.243911 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:31.243887 2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 19:38:31.244039 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:31.243932 2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 19:38:31.356557 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:31.356521 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" event={"ID":"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1","Type":"ContainerStarted","Data":"21da6f7a2ea1cff642cddf37e360c8ec0193615a70b8c5b1f2a5094890a47652"} Jul 15 19:38:31.357012 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:31.356580 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mhtt6" Jul 15 19:38:31.357391 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:31.357372 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mhtt6" Jul 15 19:38:32.235731 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:32.235537 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:32.235922 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:32.235558 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:32.235922 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:32.235854 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv9rh" podUID="87560cf2-a9d6-46e4-86dd-eac6f20cc8c8" Jul 15 19:38:32.236115 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:32.235987 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ws6h" podUID="9db8df3b-ca71-41fb-9039-a9b1afc9f1bd" Jul 15 19:38:32.362250 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:32.362221 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 19:38:32.362737 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:32.362701 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" event={"ID":"16c0161e-1bfd-4b7b-93e0-a0da2acad94f","Type":"ContainerStarted","Data":"fd6280163d467e66fa232572aa4c6699fb9dca5f0569cf9defee69062a35f341"} Jul 15 19:38:33.233017 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:33.232984 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:33.233203 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:33.233156 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-477vh" podUID="33e0c74e-e752-4c0f-9c26-a9742b866f41" Jul 15 19:38:34.233308 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:34.233133 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:34.234019 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:34.233222 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:34.234019 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:34.233387 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ws6h" podUID="9db8df3b-ca71-41fb-9039-a9b1afc9f1bd" Jul 15 19:38:34.234019 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:34.233489 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv9rh" podUID="87560cf2-a9d6-46e4-86dd-eac6f20cc8c8" Jul 15 19:38:34.369833 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:34.369793 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" event={"ID":"49dbcdc8-5c27-4b5c-91e7-e3e3560bdbc1","Type":"ContainerStarted","Data":"4f2c3d590e4cf27a3f02d622a05fd88546bbfe7ddf6791b81939e717e0c8b7df"} Jul 15 19:38:34.371499 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:34.371471 2554 generic.go:356] "Generic (PLEG): container finished" podID="c865faf5-fd82-4070-b169-a018a5c0bb0d" containerID="6529d8a55c2bcbe8b414e058348c63162acaac32808fb898faac5cce847e9b3e" exitCode=0 Jul 15 19:38:34.371614 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:34.371544 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9mp58" event={"ID":"c865faf5-fd82-4070-b169-a018a5c0bb0d","Type":"ContainerDied","Data":"6529d8a55c2bcbe8b414e058348c63162acaac32808fb898faac5cce847e9b3e"} Jul 15 19:38:34.375112 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:34.375093 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 19:38:34.375449 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:34.375424 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" event={"ID":"16c0161e-1bfd-4b7b-93e0-a0da2acad94f","Type":"ContainerStarted","Data":"2100144474d887173b3f18a89e35a5d2e712a199937883d52401a4c4e2b561b2"} Jul 15 19:38:34.375702 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:34.375683 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:34.375789 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:34.375775 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:34.375848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:34.375827 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:34.375886 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:34.375826 2554 scope.go:117] "RemoveContainer" containerID="5a684503d266404317cff2ac74dddffa986c6a21eb1862249eb7ab73b534541f" Jul 15 19:38:34.389484 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:34.389441 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lswmx" podStartSLOduration=4.287936428 podStartE2EDuration="28.389426345s" totalImagesPullingTime="24.101489917s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:38:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:34.389375156 +0000 UTC m=+28.633881629" watchObservedRunningTime="2026-07-15 19:38:34.389426345 +0000 UTC m=+28.633932805" Jul 15 19:38:34.393085 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:34.393063 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:34.393198 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:34.393170 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:38:35.232998 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:35.232965 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:35.233247 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:35.233096 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-477vh" podUID="33e0c74e-e752-4c0f-9c26-a9742b866f41" Jul 15 19:38:35.382473 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:35.382449 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 19:38:35.382914 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:35.382887 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" event={"ID":"16c0161e-1bfd-4b7b-93e0-a0da2acad94f","Type":"ContainerStarted","Data":"f15c129d72d2eb09666f1fdcd92a28b3458d284bf3d3e214cfee7067522cf27e"} Jul 15 19:38:35.416239 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:35.416184 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" podStartSLOduration=9.84805505 podStartE2EDuration="29.416163071s" totalImagesPullingTime="19.568108021s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:35.41438972 +0000 UTC m=+29.658896180" watchObservedRunningTime="2026-07-15 19:38:35.416163071 +0000 UTC m=+29.660669533" Jul 15 19:38:35.822699 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:35.822470 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2ws6h"] Jul 15 19:38:35.823031 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:35.822774 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:35.823031 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:35.822866 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ws6h" podUID="9db8df3b-ca71-41fb-9039-a9b1afc9f1bd" Jul 15 19:38:35.826349 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:35.826316 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-477vh"] Jul 15 19:38:35.826497 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:35.826441 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:35.826582 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:35.826559 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-477vh" podUID="33e0c74e-e752-4c0f-9c26-a9742b866f41" Jul 15 19:38:35.826997 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:35.826975 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rv9rh"] Jul 15 19:38:35.827112 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:35.827057 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:35.827162 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:35.827134 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv9rh" podUID="87560cf2-a9d6-46e4-86dd-eac6f20cc8c8" Jul 15 19:38:36.387389 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:36.387352 2554 generic.go:356] "Generic (PLEG): container finished" podID="c865faf5-fd82-4070-b169-a018a5c0bb0d" containerID="fef566af29d34056389f0d1a4e92ee4c68308d564d15cef5d0f837dee35e4b25" exitCode=0 Jul 15 19:38:36.387783 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:36.387438 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9mp58" event={"ID":"c865faf5-fd82-4070-b169-a018a5c0bb0d","Type":"ContainerDied","Data":"fef566af29d34056389f0d1a4e92ee4c68308d564d15cef5d0f837dee35e4b25"} Jul 15 19:38:37.232939 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:37.232906 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:37.233148 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:37.233008 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:37.233148 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:37.233018 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv9rh" podUID="87560cf2-a9d6-46e4-86dd-eac6f20cc8c8" Jul 15 19:38:37.233148 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:37.233037 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:37.233148 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:37.233115 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-477vh" podUID="33e0c74e-e752-4c0f-9c26-a9742b866f41" Jul 15 19:38:37.233311 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:37.233182 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ws6h" podUID="9db8df3b-ca71-41fb-9039-a9b1afc9f1bd" Jul 15 19:38:38.395044 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:38.395005 2554 generic.go:356] "Generic (PLEG): container finished" podID="c865faf5-fd82-4070-b169-a018a5c0bb0d" containerID="9861708ee11643ee4df7d48c5cc534144bbdc11687fa56d8856a4bbacbfbb9ae" exitCode=0 Jul 15 19:38:38.395699 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:38.395061 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9mp58" event={"ID":"c865faf5-fd82-4070-b169-a018a5c0bb0d","Type":"ContainerDied","Data":"9861708ee11643ee4df7d48c5cc534144bbdc11687fa56d8856a4bbacbfbb9ae"} Jul 15 19:38:39.232741 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:39.232707 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:39.232947 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:39.232772 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:39.232947 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:39.232771 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:39.232947 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:39.232861 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv9rh" podUID="87560cf2-a9d6-46e4-86dd-eac6f20cc8c8" Jul 15 19:38:39.233120 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:39.232951 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ws6h" podUID="9db8df3b-ca71-41fb-9039-a9b1afc9f1bd" Jul 15 19:38:39.233120 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:39.233061 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-477vh" podUID="33e0c74e-e752-4c0f-9c26-a9742b866f41" Jul 15 19:38:39.924845 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:39.924805 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs\") pod \"network-metrics-daemon-477vh\" (UID: \"33e0c74e-e752-4c0f-9c26-a9742b866f41\") " pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:39.925473 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:39.924859 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret\") pod \"global-pull-secret-syncer-rv9rh\" (UID: \"87560cf2-a9d6-46e4-86dd-eac6f20cc8c8\") " pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:39.925473 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:39.924975 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:38:39.925473 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:39.924983 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:38:39.925473 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:39.925043 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret podName:87560cf2-a9d6-46e4-86dd-eac6f20cc8c8 nodeName:}" failed. No retries permitted until 2026-07-15 19:39:11.925024397 +0000 UTC m=+66.169530838 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret") pod "global-pull-secret-syncer-rv9rh" (UID: "87560cf2-a9d6-46e4-86dd-eac6f20cc8c8") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:38:39.925473 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:39.925061 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs podName:33e0c74e-e752-4c0f-9c26-a9742b866f41 nodeName:}" failed. No retries permitted until 2026-07-15 19:39:11.925051156 +0000 UTC m=+66.169557595 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs") pod "network-metrics-daemon-477vh" (UID: "33e0c74e-e752-4c0f-9c26-a9742b866f41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:38:40.025445 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:40.025414 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzvq\" (UniqueName: \"kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq\") pod \"network-check-target-2ws6h\" (UID: \"9db8df3b-ca71-41fb-9039-a9b1afc9f1bd\") " pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:40.025602 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:40.025541 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:38:40.025602 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:40.025560 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:38:40.025602 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:40.025573 2554 projected.go:196] Error preparing data for projected volume kube-api-access-jfzvq for pod openshift-network-diagnostics/network-check-target-2ws6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:38:40.025804 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:40.025629 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq podName:9db8df3b-ca71-41fb-9039-a9b1afc9f1bd nodeName:}" failed. No retries permitted until 2026-07-15 19:39:12.025611914 +0000 UTC m=+66.270118373 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfzvq" (UniqueName: "kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq") pod "network-check-target-2ws6h" (UID: "9db8df3b-ca71-41fb-9039-a9b1afc9f1bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:38:41.232335 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:41.232302 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:41.232743 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:41.232399 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:41.232743 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:41.232409 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv9rh" podUID="87560cf2-a9d6-46e4-86dd-eac6f20cc8c8" Jul 15 19:38:41.232743 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:41.232430 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:41.232743 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:41.232510 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-477vh" podUID="33e0c74e-e752-4c0f-9c26-a9742b866f41" Jul 15 19:38:41.232743 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:41.232561 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2ws6h" podUID="9db8df3b-ca71-41fb-9039-a9b1afc9f1bd" Jul 15 19:38:42.598822 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.598796 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-139.ec2.internal" event="NodeReady" Jul 15 19:38:42.599114 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.598926 2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 19:38:42.642108 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.641876 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7"] Jul 15 19:38:42.645170 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.645145 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-2ldzn"] Jul 15 19:38:42.645309 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.645233 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7" Jul 15 19:38:42.646962 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.646909 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-pmncb"] Jul 15 19:38:42.647086 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.646990 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-2ldzn" Jul 15 19:38:42.647601 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.647582 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 15 19:38:42.648351 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.648311 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d66974864-7mgdf"] Jul 15 19:38:42.648417 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.648363 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:38:42.648552 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.648509 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:42.649946 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.649929 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftq5p"] Jul 15 19:38:42.650067 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.650051 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.650695 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.650680 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ml8fw\"" Jul 15 19:38:42.651312 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.651107 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 15 19:38:42.651312 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.651152 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:38:42.651445 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.651185 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 15 19:38:42.651530 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.651511 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2"] Jul 15 19:38:42.651608 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.651583 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 15 19:38:42.651608 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.651597 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 15 19:38:42.651799 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.651782 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tlm8p\"" Jul 15 19:38:42.652152 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.652133 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftq5p" Jul 15 19:38:42.652251 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.652177 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-859v6\"" Jul 15 19:38:42.652251 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.652239 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 19:38:42.652373 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.652260 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 15 19:38:42.652452 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.652142 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 19:38:42.652506 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.652443 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 15 19:38:42.653259 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.653241 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-69x22"] Jul 15 19:38:42.653471 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.653452 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 15 19:38:42.653777 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.653649 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 15 19:38:42.653967 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.653947 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" Jul 15 19:38:42.654389 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.654370 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 15 19:38:42.654471 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.654398 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2tjcx\"" Jul 15 19:38:42.655191 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.655173 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-656f8d6d5b-5lzhw"] Jul 15 19:38:42.655559 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.655298 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-69x22" Jul 15 19:38:42.660403 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.660382 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 15 19:38:42.660403 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.660397 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 15 19:38:42.660539 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.660384 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-q8blt\"" Jul 15 19:38:42.660717 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.660684 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 15 19:38:42.660796 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.660725 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:38:42.660796 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.660777 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 19:38:42.660906 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.660802 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 19:38:42.660906 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.660866 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zms8d"] Jul 15 19:38:42.661048 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.660964 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 15 19:38:42.661048 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.660976 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:42.661650 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.661632 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 15 19:38:42.661824 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.661809 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 15 19:38:42.661896 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.661835 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vdnlr\"" Jul 15 19:38:42.662624 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.662600 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zms8d" Jul 15 19:38:42.664810 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.664794 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 15 19:38:42.665960 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.665939 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 15 19:38:42.666582 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.666483 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 15 19:38:42.678132 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.678105 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 15 19:38:42.678233 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.678169 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:38:42.678435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.678415 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 19:38:42.678598 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.678569 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 15 19:38:42.678735 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.678719 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 15 19:38:42.683965 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.683946 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7"] Jul 15 19:38:42.685801 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.685405 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-69x22"] Jul 15 19:38:42.686810 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.686786 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-2ldzn"] Jul 15 19:38:42.687407 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.687388 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2f2ll\"" Jul 15 19:38:42.687978 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.687556 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 15 19:38:42.687978 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.687655 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d66974864-7mgdf"] Jul 15 19:38:42.688195 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.688009 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 19:38:42.688377 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.688359 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-pmncb"] Jul 15 19:38:42.688512 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.688496 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ngg2l\"" Jul 15 19:38:42.688906 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.688890 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-bfzzn\"" Jul 15 19:38:42.689099 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.689085 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-ffl9z"] Jul 15 19:38:42.690647 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.689583 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 15 19:38:42.690647 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.689658 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 15 19:38:42.690647 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.689672 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:38:42.690647 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.689886 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 15 19:38:42.691602 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.691499 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-k8brr"] Jul 15 19:38:42.691678 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.691659 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ffl9z" Jul 15 19:38:42.693830 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.693811 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2"] Jul 15 19:38:42.693974 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.693959 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-k8brr" Jul 15 19:38:42.694853 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.694608 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 19:38:42.694853 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.694670 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-97crw\"" Jul 15 19:38:42.694853 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.694804 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-k8brr"] Jul 15 19:38:42.694853 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.694847 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 19:38:42.697396 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.697201 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 15 19:38:42.699780 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.699678 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-mlvd8\"" Jul 15 19:38:42.700139 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.699957 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 15 19:38:42.701037 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.700942 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftq5p"] Jul 15 19:38:42.702648 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.702630 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zms8d"] Jul 15 19:38:42.715116 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.715087 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-656f8d6d5b-5lzhw"] Jul 15 19:38:42.717276 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.717252 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-ffl9z"] Jul 15 19:38:42.718307 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.718286 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2288d"] Jul 15 19:38:42.720153 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.720137 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zgrjv"] Jul 15 19:38:42.720305 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.720283 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2288d" Jul 15 19:38:42.721989 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.721876 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zgrjv" Jul 15 19:38:42.723055 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.723037 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-t8bxm\"" Jul 15 19:38:42.723165 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.723035 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 19:38:42.723165 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.723090 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 19:38:42.723270 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.723094 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 19:38:42.724925 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.724911 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2fvq2\"" Jul 15 19:38:42.725780 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.725767 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 19:38:42.726319 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.726306 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 19:38:42.731647 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.731613 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zgrjv"] Jul 15 19:38:42.733046 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.732803 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2288d"] Jul 15 19:38:42.746107 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746085 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b90b66a6-7fd1-47c8-8250-83f56348498c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-957d7\" (UID: \"b90b66a6-7fd1-47c8-8250-83f56348498c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7" Jul 15 19:38:42.746208 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746120 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b21f8771-ffd5-4487-812b-bfbaa7718461-serving-cert\") pod \"console-operator-7866497494-2ldzn\" (UID: \"b21f8771-ffd5-4487-812b-bfbaa7718461\") " pod="openshift-console-operator/console-operator-7866497494-2ldzn" Jul 15 19:38:42.746208 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746151 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj2cg\" (UniqueName: \"kubernetes.io/projected/eb9ebe84-37a6-4f73-880e-dad90c6453c7-kube-api-access-tj2cg\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:42.746208 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746192 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-default-certificate\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:42.746327 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746217 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2691146-bc04-4c7b-bb0c-b3377762da6e-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-pmncb\" (UID: \"e2691146-bc04-4c7b-bb0c-b3377762da6e\") " pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:42.746327 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746260 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-stats-auth\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:42.746327 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746280 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9slz\" (UniqueName: \"kubernetes.io/projected/2ae8e73f-008e-4977-8153-342f4a7460a5-kube-api-access-b9slz\") pod \"kube-storage-version-migrator-operator-799cb7f958-69x22\" (UID: \"2ae8e73f-008e-4977-8153-342f4a7460a5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-69x22" Jul 15 19:38:42.746327 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746299 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2691146-bc04-4c7b-bb0c-b3377762da6e-serving-cert\") pod \"insights-operator-68d9dd94cf-pmncb\" (UID: \"e2691146-bc04-4c7b-bb0c-b3377762da6e\") " pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:42.746327 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746324 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-certificates\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.746532 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746343 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69vlc\" (UniqueName: \"kubernetes.io/projected/e3227b7b-2925-413d-aaf3-aa5e5bb60a81-kube-api-access-69vlc\") pod \"service-ca-operator-7695fc6c8c-ftq5p\" (UID: \"e3227b7b-2925-413d-aaf3-aa5e5bb60a81\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftq5p" Jul 15 19:38:42.746532 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746364 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b21f8771-ffd5-4487-812b-bfbaa7718461-trusted-ca\") pod \"console-operator-7866497494-2ldzn\" (UID: \"b21f8771-ffd5-4487-812b-bfbaa7718461\") " pod="openshift-console-operator/console-operator-7866497494-2ldzn" Jul 15 19:38:42.746532 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746386 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3227b7b-2925-413d-aaf3-aa5e5bb60a81-serving-cert\") pod \"service-ca-operator-7695fc6c8c-ftq5p\" (UID: \"e3227b7b-2925-413d-aaf3-aa5e5bb60a81\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftq5p" Jul 15 19:38:42.746532 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746413 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8s86\" (UniqueName: \"kubernetes.io/projected/b90b66a6-7fd1-47c8-8250-83f56348498c-kube-api-access-s8s86\") pod \"cluster-samples-operator-65db858894-957d7\" (UID: \"b90b66a6-7fd1-47c8-8250-83f56348498c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7" Jul 15 19:38:42.746532 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746441 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e2691146-bc04-4c7b-bb0c-b3377762da6e-snapshots\") pod \"insights-operator-68d9dd94cf-pmncb\" (UID: \"e2691146-bc04-4c7b-bb0c-b3377762da6e\") " pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:42.746532 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746478 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.746532 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746516 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-fksq2\" (UID: \"88a9eb45-3875-4d7f-aab1-e1aeb90333d6\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" Jul 15 19:38:42.746855 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746543 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3227b7b-2925-413d-aaf3-aa5e5bb60a81-config\") pod \"service-ca-operator-7695fc6c8c-ftq5p\" (UID: \"e3227b7b-2925-413d-aaf3-aa5e5bb60a81\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftq5p" Jul 15 19:38:42.746855 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746572 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-fksq2\" (UID: \"88a9eb45-3875-4d7f-aab1-e1aeb90333d6\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" Jul 15 19:38:42.746855 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746594 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:42.746855 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746610 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2ae8e73f-008e-4977-8153-342f4a7460a5-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-69x22\" (UID: \"2ae8e73f-008e-4977-8153-342f4a7460a5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-69x22" Jul 15 19:38:42.746855 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746636 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72b9d425-51d4-4da9-8d59-bd071ec852bb-trusted-ca\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.746855 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746678 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/72b9d425-51d4-4da9-8d59-bd071ec852bb-image-registry-private-configuration\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.746855 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746705 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/72b9d425-51d4-4da9-8d59-bd071ec852bb-ca-trust-extracted\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.746855 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746728 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b21f8771-ffd5-4487-812b-bfbaa7718461-config\") pod \"console-operator-7866497494-2ldzn\" (UID: \"b21f8771-ffd5-4487-812b-bfbaa7718461\") " pod="openshift-console-operator/console-operator-7866497494-2ldzn" Jul 15 19:38:42.746855 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746774 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2691146-bc04-4c7b-bb0c-b3377762da6e-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-pmncb\" (UID: \"e2691146-bc04-4c7b-bb0c-b3377762da6e\") " pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:42.746855 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746802 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:42.746855 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746843 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ae8e73f-008e-4977-8153-342f4a7460a5-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-69x22\" (UID: \"2ae8e73f-008e-4977-8153-342f4a7460a5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-69x22" Jul 15 19:38:42.747248 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746872 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmwvb\" (UniqueName: \"kubernetes.io/projected/e2691146-bc04-4c7b-bb0c-b3377762da6e-kube-api-access-rmwvb\") pod \"insights-operator-68d9dd94cf-pmncb\" (UID: \"e2691146-bc04-4c7b-bb0c-b3377762da6e\") " pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:42.747248 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746906 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/72b9d425-51d4-4da9-8d59-bd071ec852bb-installation-pull-secrets\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.747248 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746950 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-bound-sa-token\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.747248 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746970 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvj9g\" (UniqueName: \"kubernetes.io/projected/b21f8771-ffd5-4487-812b-bfbaa7718461-kube-api-access-dvj9g\") pod \"console-operator-7866497494-2ldzn\" (UID: \"b21f8771-ffd5-4487-812b-bfbaa7718461\") " pod="openshift-console-operator/console-operator-7866497494-2ldzn" Jul 15 19:38:42.747248 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.746987 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j7sk\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-kube-api-access-7j7sk\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.747248 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.747013 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2691146-bc04-4c7b-bb0c-b3377762da6e-tmp\") pod \"insights-operator-68d9dd94cf-pmncb\" (UID: \"e2691146-bc04-4c7b-bb0c-b3377762da6e\") " pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:42.747248 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.747029 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsmhs\" (UniqueName: \"kubernetes.io/projected/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-kube-api-access-zsmhs\") pod \"cluster-monitoring-operator-5f7f494dd4-fksq2\" (UID: \"88a9eb45-3875-4d7f-aab1-e1aeb90333d6\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" Jul 15 19:38:42.747248 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.747045 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlkqj\" (UniqueName: \"kubernetes.io/projected/99bcfa5f-a482-4e96-bc3f-a6e3365078d3-kube-api-access-hlkqj\") pod \"volume-data-source-validator-78495597f6-zms8d\" (UID: \"99bcfa5f-a482-4e96-bc3f-a6e3365078d3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zms8d" Jul 15 19:38:42.848206 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848098 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86437276-af54-49be-acf6-91f82d55e92c-tmp-dir\") pod \"dns-default-zgrjv\" (UID: \"86437276-af54-49be-acf6-91f82d55e92c\") " pod="openshift-dns/dns-default-zgrjv" Jul 15 19:38:42.848206 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848135 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-fksq2\" (UID: \"88a9eb45-3875-4d7f-aab1-e1aeb90333d6\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" Jul 15 19:38:42.848206 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848153 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3227b7b-2925-413d-aaf3-aa5e5bb60a81-config\") pod \"service-ca-operator-7695fc6c8c-ftq5p\" (UID: \"e3227b7b-2925-413d-aaf3-aa5e5bb60a81\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftq5p" Jul 15 19:38:42.848206 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848195 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-fksq2\" (UID: \"88a9eb45-3875-4d7f-aab1-e1aeb90333d6\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" Jul 15 19:38:42.848510 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848225 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:42.848510 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848242 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2ae8e73f-008e-4977-8153-342f4a7460a5-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-69x22\" (UID: \"2ae8e73f-008e-4977-8153-342f4a7460a5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-69x22" Jul 15 19:38:42.848510 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848258 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72b9d425-51d4-4da9-8d59-bd071ec852bb-trusted-ca\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.848510 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848283 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/72b9d425-51d4-4da9-8d59-bd071ec852bb-image-registry-private-configuration\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.848510 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848306 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/72b9d425-51d4-4da9-8d59-bd071ec852bb-ca-trust-extracted\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.848510 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:42.848336 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 15 19:38:42.848510 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:42.848414 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle podName:eb9ebe84-37a6-4f73-880e-dad90c6453c7 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:43.348388921 +0000 UTC m=+37.592895381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle") pod "router-default-656f8d6d5b-5lzhw" (UID: "eb9ebe84-37a6-4f73-880e-dad90c6453c7") : configmap references non-existent config key: service-ca.crt Jul 15 19:38:42.848510 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:42.848448 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls podName:88a9eb45-3875-4d7f-aab1-e1aeb90333d6 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:43.348430204 +0000 UTC m=+37.592936662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-fksq2" (UID: "88a9eb45-3875-4d7f-aab1-e1aeb90333d6") : secret "cluster-monitoring-operator-tls" not found Jul 15 19:38:42.848510 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848493 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b21f8771-ffd5-4487-812b-bfbaa7718461-config\") pod \"console-operator-7866497494-2ldzn\" (UID: \"b21f8771-ffd5-4487-812b-bfbaa7718461\") " pod="openshift-console-operator/console-operator-7866497494-2ldzn" Jul 15 19:38:42.849070 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848529 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2691146-bc04-4c7b-bb0c-b3377762da6e-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-pmncb\" (UID: \"e2691146-bc04-4c7b-bb0c-b3377762da6e\") " pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:42.849070 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848556 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:42.849070 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848588 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5897\" (UniqueName: \"kubernetes.io/projected/d896f9b8-775c-4130-820c-59831ff12641-kube-api-access-g5897\") pod \"ingress-canary-2288d\" (UID: \"d896f9b8-775c-4130-820c-59831ff12641\") " pod="openshift-ingress-canary/ingress-canary-2288d" Jul 15 19:38:42.849070 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848635 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ae8e73f-008e-4977-8153-342f4a7460a5-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-69x22\" (UID: \"2ae8e73f-008e-4977-8153-342f4a7460a5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-69x22" Jul 15 19:38:42.849070 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848667 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmwvb\" (UniqueName: \"kubernetes.io/projected/e2691146-bc04-4c7b-bb0c-b3377762da6e-kube-api-access-rmwvb\") pod \"insights-operator-68d9dd94cf-pmncb\" (UID: \"e2691146-bc04-4c7b-bb0c-b3377762da6e\") " pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:42.849070 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848705 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8tlt\" (UniqueName: \"kubernetes.io/projected/86437276-af54-49be-acf6-91f82d55e92c-kube-api-access-l8tlt\") pod \"dns-default-zgrjv\" (UID: \"86437276-af54-49be-acf6-91f82d55e92c\") " pod="openshift-dns/dns-default-zgrjv" Jul 15 19:38:42.849070 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848727 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3227b7b-2925-413d-aaf3-aa5e5bb60a81-config\") pod \"service-ca-operator-7695fc6c8c-ftq5p\" (UID: \"e3227b7b-2925-413d-aaf3-aa5e5bb60a81\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftq5p" Jul 15 19:38:42.849070 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848737 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/72b9d425-51d4-4da9-8d59-bd071ec852bb-installation-pull-secrets\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.849070 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848794 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-bound-sa-token\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.849070 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848821 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvj9g\" (UniqueName: \"kubernetes.io/projected/b21f8771-ffd5-4487-812b-bfbaa7718461-kube-api-access-dvj9g\") pod \"console-operator-7866497494-2ldzn\" (UID: \"b21f8771-ffd5-4487-812b-bfbaa7718461\") " pod="openshift-console-operator/console-operator-7866497494-2ldzn" Jul 15 19:38:42.849070 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848853 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j7sk\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-kube-api-access-7j7sk\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.849070 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848880 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls\") pod \"dns-default-zgrjv\" (UID: \"86437276-af54-49be-acf6-91f82d55e92c\") " pod="openshift-dns/dns-default-zgrjv" Jul 15 19:38:42.849070 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848908 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpz6r\" (UniqueName: \"kubernetes.io/projected/49cd2b3f-c1fa-4cf8-a1b8-5b5583414a9b-kube-api-access-bpz6r\") pod \"network-check-source-869dcfcdcd-ffl9z\" (UID: \"49cd2b3f-c1fa-4cf8-a1b8-5b5583414a9b\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ffl9z" Jul 15 19:38:42.849070 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848933 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-nginx-conf\") pod \"networking-console-plugin-6cf856758c-k8brr\" (UID: \"7a0eeb16-5201-43e8-a850-a7a3cdedc3a1\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k8brr" Jul 15 19:38:42.849070 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.848976 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2691146-bc04-4c7b-bb0c-b3377762da6e-tmp\") pod \"insights-operator-68d9dd94cf-pmncb\" (UID: \"e2691146-bc04-4c7b-bb0c-b3377762da6e\") " pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:42.849070 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:42.848985 2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 19:38:42.849873 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849001 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsmhs\" (UniqueName: \"kubernetes.io/projected/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-kube-api-access-zsmhs\") pod \"cluster-monitoring-operator-5f7f494dd4-fksq2\" (UID: \"88a9eb45-3875-4d7f-aab1-e1aeb90333d6\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" Jul 15 19:38:42.849873 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849013 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-fksq2\" (UID: \"88a9eb45-3875-4d7f-aab1-e1aeb90333d6\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" Jul 15 19:38:42.849873 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849032 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlkqj\" (UniqueName: \"kubernetes.io/projected/99bcfa5f-a482-4e96-bc3f-a6e3365078d3-kube-api-access-hlkqj\") pod \"volume-data-source-validator-78495597f6-zms8d\" (UID: \"99bcfa5f-a482-4e96-bc3f-a6e3365078d3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zms8d" Jul 15 19:38:42.849873 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849085 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b90b66a6-7fd1-47c8-8250-83f56348498c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-957d7\" (UID: \"b90b66a6-7fd1-47c8-8250-83f56348498c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7" Jul 15 19:38:42.849873 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849115 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b21f8771-ffd5-4487-812b-bfbaa7718461-serving-cert\") pod \"console-operator-7866497494-2ldzn\" (UID: \"b21f8771-ffd5-4487-812b-bfbaa7718461\") " pod="openshift-console-operator/console-operator-7866497494-2ldzn" Jul 15 19:38:42.849873 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849153 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86437276-af54-49be-acf6-91f82d55e92c-config-volume\") pod \"dns-default-zgrjv\" (UID: \"86437276-af54-49be-acf6-91f82d55e92c\") " pod="openshift-dns/dns-default-zgrjv" Jul 15 19:38:42.849873 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849179 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-k8brr\" (UID: \"7a0eeb16-5201-43e8-a850-a7a3cdedc3a1\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k8brr" Jul 15 19:38:42.849873 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:42.849205 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs podName:eb9ebe84-37a6-4f73-880e-dad90c6453c7 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:43.349190529 +0000 UTC m=+37.593696970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs") pod "router-default-656f8d6d5b-5lzhw" (UID: "eb9ebe84-37a6-4f73-880e-dad90c6453c7") : secret "router-metrics-certs-default" not found Jul 15 19:38:42.849873 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849242 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj2cg\" (UniqueName: \"kubernetes.io/projected/eb9ebe84-37a6-4f73-880e-dad90c6453c7-kube-api-access-tj2cg\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:42.849873 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849276 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-default-certificate\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:42.849873 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849302 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2691146-bc04-4c7b-bb0c-b3377762da6e-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-pmncb\" (UID: \"e2691146-bc04-4c7b-bb0c-b3377762da6e\") " pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:42.849873 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849331 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-stats-auth\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:42.849873 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849360 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9slz\" (UniqueName: \"kubernetes.io/projected/2ae8e73f-008e-4977-8153-342f4a7460a5-kube-api-access-b9slz\") pod \"kube-storage-version-migrator-operator-799cb7f958-69x22\" (UID: \"2ae8e73f-008e-4977-8153-342f4a7460a5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-69x22" Jul 15 19:38:42.849873 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849385 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2691146-bc04-4c7b-bb0c-b3377762da6e-serving-cert\") pod \"insights-operator-68d9dd94cf-pmncb\" (UID: \"e2691146-bc04-4c7b-bb0c-b3377762da6e\") " pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:42.849873 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849410 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-certificates\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.850906 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849418 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b21f8771-ffd5-4487-812b-bfbaa7718461-config\") pod \"console-operator-7866497494-2ldzn\" (UID: \"b21f8771-ffd5-4487-812b-bfbaa7718461\") " pod="openshift-console-operator/console-operator-7866497494-2ldzn" Jul 15 19:38:42.850906 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849439 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69vlc\" (UniqueName: \"kubernetes.io/projected/e3227b7b-2925-413d-aaf3-aa5e5bb60a81-kube-api-access-69vlc\") pod \"service-ca-operator-7695fc6c8c-ftq5p\" (UID: \"e3227b7b-2925-413d-aaf3-aa5e5bb60a81\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftq5p" Jul 15 19:38:42.850906 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849474 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b21f8771-ffd5-4487-812b-bfbaa7718461-trusted-ca\") pod \"console-operator-7866497494-2ldzn\" (UID: \"b21f8771-ffd5-4487-812b-bfbaa7718461\") " pod="openshift-console-operator/console-operator-7866497494-2ldzn" Jul 15 19:38:42.850906 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849497 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3227b7b-2925-413d-aaf3-aa5e5bb60a81-serving-cert\") pod \"service-ca-operator-7695fc6c8c-ftq5p\" (UID: \"e3227b7b-2925-413d-aaf3-aa5e5bb60a81\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftq5p" Jul 15 19:38:42.850906 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849536 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d896f9b8-775c-4130-820c-59831ff12641-cert\") pod \"ingress-canary-2288d\" (UID: \"d896f9b8-775c-4130-820c-59831ff12641\") " pod="openshift-ingress-canary/ingress-canary-2288d" Jul 15 19:38:42.850906 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849570 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8s86\" (UniqueName: \"kubernetes.io/projected/b90b66a6-7fd1-47c8-8250-83f56348498c-kube-api-access-s8s86\") pod \"cluster-samples-operator-65db858894-957d7\" (UID: \"b90b66a6-7fd1-47c8-8250-83f56348498c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7" Jul 15 19:38:42.850906 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849595 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e2691146-bc04-4c7b-bb0c-b3377762da6e-snapshots\") pod \"insights-operator-68d9dd94cf-pmncb\" (UID: \"e2691146-bc04-4c7b-bb0c-b3377762da6e\") " pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:42.850906 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849627 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.850906 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:42.849777 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 19:38:42.850906 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:42.849793 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d66974864-7mgdf: secret "image-registry-tls" not found Jul 15 19:38:42.850906 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:42.849854 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls podName:72b9d425-51d4-4da9-8d59-bd071ec852bb nodeName:}" failed. No retries permitted until 2026-07-15 19:38:43.349840621 +0000 UTC m=+37.594347061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls") pod "image-registry-6d66974864-7mgdf" (UID: "72b9d425-51d4-4da9-8d59-bd071ec852bb") : secret "image-registry-tls" not found Jul 15 19:38:42.850906 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849917 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ae8e73f-008e-4977-8153-342f4a7460a5-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-69x22\" (UID: \"2ae8e73f-008e-4977-8153-342f4a7460a5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-69x22" Jul 15 19:38:42.850906 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.850181 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2691146-bc04-4c7b-bb0c-b3377762da6e-tmp\") pod \"insights-operator-68d9dd94cf-pmncb\" (UID: \"e2691146-bc04-4c7b-bb0c-b3377762da6e\") " pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:42.850906 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:42.850245 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 15 19:38:42.850906 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:42.850291 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b90b66a6-7fd1-47c8-8250-83f56348498c-samples-operator-tls podName:b90b66a6-7fd1-47c8-8250-83f56348498c nodeName:}" failed. No retries permitted until 2026-07-15 19:38:43.350276939 +0000 UTC m=+37.594783378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b90b66a6-7fd1-47c8-8250-83f56348498c-samples-operator-tls") pod "cluster-samples-operator-65db858894-957d7" (UID: "b90b66a6-7fd1-47c8-8250-83f56348498c") : secret "samples-operator-tls" not found Jul 15 19:38:42.851712 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.851134 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-certificates\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.851712 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.851297 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/72b9d425-51d4-4da9-8d59-bd071ec852bb-ca-trust-extracted\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.851712 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.849494 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2691146-bc04-4c7b-bb0c-b3377762da6e-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-pmncb\" (UID: \"e2691146-bc04-4c7b-bb0c-b3377762da6e\") " pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:42.851712 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.851444 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72b9d425-51d4-4da9-8d59-bd071ec852bb-trusted-ca\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.851948 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.851896 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e2691146-bc04-4c7b-bb0c-b3377762da6e-snapshots\") pod \"insights-operator-68d9dd94cf-pmncb\" (UID: \"e2691146-bc04-4c7b-bb0c-b3377762da6e\") " pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:42.851948 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.851918 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b21f8771-ffd5-4487-812b-bfbaa7718461-trusted-ca\") pod \"console-operator-7866497494-2ldzn\" (UID: \"b21f8771-ffd5-4487-812b-bfbaa7718461\") " pod="openshift-console-operator/console-operator-7866497494-2ldzn" Jul 15 19:38:42.852396 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.852346 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2691146-bc04-4c7b-bb0c-b3377762da6e-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-pmncb\" (UID: \"e2691146-bc04-4c7b-bb0c-b3377762da6e\") " pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:42.853567 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.853539 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2ae8e73f-008e-4977-8153-342f4a7460a5-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-69x22\" (UID: \"2ae8e73f-008e-4977-8153-342f4a7460a5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-69x22" Jul 15 19:38:42.853700 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.853679 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/72b9d425-51d4-4da9-8d59-bd071ec852bb-installation-pull-secrets\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.854061 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.854037 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3227b7b-2925-413d-aaf3-aa5e5bb60a81-serving-cert\") pod \"service-ca-operator-7695fc6c8c-ftq5p\" (UID: \"e3227b7b-2925-413d-aaf3-aa5e5bb60a81\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftq5p" Jul 15 19:38:42.854153 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.854092 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b21f8771-ffd5-4487-812b-bfbaa7718461-serving-cert\") pod \"console-operator-7866497494-2ldzn\" (UID: \"b21f8771-ffd5-4487-812b-bfbaa7718461\") " pod="openshift-console-operator/console-operator-7866497494-2ldzn" Jul 15 19:38:42.854262 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.854242 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-stats-auth\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:42.854506 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.854486 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/72b9d425-51d4-4da9-8d59-bd071ec852bb-image-registry-private-configuration\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.854540 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.854504 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-default-certificate\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:42.854841 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.854826 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2691146-bc04-4c7b-bb0c-b3377762da6e-serving-cert\") pod \"insights-operator-68d9dd94cf-pmncb\" (UID: \"e2691146-bc04-4c7b-bb0c-b3377762da6e\") " pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:42.864982 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.864956 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmwvb\" (UniqueName: \"kubernetes.io/projected/e2691146-bc04-4c7b-bb0c-b3377762da6e-kube-api-access-rmwvb\") pod \"insights-operator-68d9dd94cf-pmncb\" (UID: \"e2691146-bc04-4c7b-bb0c-b3377762da6e\") " pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:42.868895 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.868875 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69vlc\" (UniqueName: \"kubernetes.io/projected/e3227b7b-2925-413d-aaf3-aa5e5bb60a81-kube-api-access-69vlc\") pod \"service-ca-operator-7695fc6c8c-ftq5p\" (UID: \"e3227b7b-2925-413d-aaf3-aa5e5bb60a81\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftq5p" Jul 15 19:38:42.869627 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.869202 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlkqj\" (UniqueName: \"kubernetes.io/projected/99bcfa5f-a482-4e96-bc3f-a6e3365078d3-kube-api-access-hlkqj\") pod \"volume-data-source-validator-78495597f6-zms8d\" (UID: \"99bcfa5f-a482-4e96-bc3f-a6e3365078d3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zms8d" Jul 15 19:38:42.869627 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.869402 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-bound-sa-token\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.869627 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.869472 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9slz\" (UniqueName: \"kubernetes.io/projected/2ae8e73f-008e-4977-8153-342f4a7460a5-kube-api-access-b9slz\") pod \"kube-storage-version-migrator-operator-799cb7f958-69x22\" (UID: \"2ae8e73f-008e-4977-8153-342f4a7460a5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-69x22" Jul 15 19:38:42.870607 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.870581 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj2cg\" (UniqueName: \"kubernetes.io/projected/eb9ebe84-37a6-4f73-880e-dad90c6453c7-kube-api-access-tj2cg\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:42.870957 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.870941 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8s86\" (UniqueName: \"kubernetes.io/projected/b90b66a6-7fd1-47c8-8250-83f56348498c-kube-api-access-s8s86\") pod \"cluster-samples-operator-65db858894-957d7\" (UID: \"b90b66a6-7fd1-47c8-8250-83f56348498c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7" Jul 15 19:38:42.871111 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.871090 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsmhs\" (UniqueName: \"kubernetes.io/projected/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-kube-api-access-zsmhs\") pod \"cluster-monitoring-operator-5f7f494dd4-fksq2\" (UID: \"88a9eb45-3875-4d7f-aab1-e1aeb90333d6\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" Jul 15 19:38:42.871344 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.871326 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j7sk\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-kube-api-access-7j7sk\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:42.871936 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.871913 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvj9g\" (UniqueName: \"kubernetes.io/projected/b21f8771-ffd5-4487-812b-bfbaa7718461-kube-api-access-dvj9g\") pod \"console-operator-7866497494-2ldzn\" (UID: \"b21f8771-ffd5-4487-812b-bfbaa7718461\") " pod="openshift-console-operator/console-operator-7866497494-2ldzn" Jul 15 19:38:42.951025 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.950992 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d896f9b8-775c-4130-820c-59831ff12641-cert\") pod \"ingress-canary-2288d\" (UID: \"d896f9b8-775c-4130-820c-59831ff12641\") " pod="openshift-ingress-canary/ingress-canary-2288d" Jul 15 19:38:42.951218 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.951041 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86437276-af54-49be-acf6-91f82d55e92c-tmp-dir\") pod \"dns-default-zgrjv\" (UID: \"86437276-af54-49be-acf6-91f82d55e92c\") " pod="openshift-dns/dns-default-zgrjv" Jul 15 19:38:42.951218 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.951206 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5897\" (UniqueName: \"kubernetes.io/projected/d896f9b8-775c-4130-820c-59831ff12641-kube-api-access-g5897\") pod \"ingress-canary-2288d\" (UID: \"d896f9b8-775c-4130-820c-59831ff12641\") " pod="openshift-ingress-canary/ingress-canary-2288d" Jul 15 19:38:42.951339 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.951256 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8tlt\" (UniqueName: \"kubernetes.io/projected/86437276-af54-49be-acf6-91f82d55e92c-kube-api-access-l8tlt\") pod \"dns-default-zgrjv\" (UID: \"86437276-af54-49be-acf6-91f82d55e92c\") " pod="openshift-dns/dns-default-zgrjv" Jul 15 19:38:42.951339 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.951297 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls\") pod \"dns-default-zgrjv\" (UID: \"86437276-af54-49be-acf6-91f82d55e92c\") " pod="openshift-dns/dns-default-zgrjv" Jul 15 19:38:42.951339 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.951327 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpz6r\" (UniqueName: \"kubernetes.io/projected/49cd2b3f-c1fa-4cf8-a1b8-5b5583414a9b-kube-api-access-bpz6r\") pod \"network-check-source-869dcfcdcd-ffl9z\" (UID: \"49cd2b3f-c1fa-4cf8-a1b8-5b5583414a9b\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ffl9z" Jul 15 19:38:42.951484 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.951354 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-nginx-conf\") pod \"networking-console-plugin-6cf856758c-k8brr\" (UID: \"7a0eeb16-5201-43e8-a850-a7a3cdedc3a1\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k8brr" Jul 15 19:38:42.951484 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.951398 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86437276-af54-49be-acf6-91f82d55e92c-tmp-dir\") pod \"dns-default-zgrjv\" (UID: \"86437276-af54-49be-acf6-91f82d55e92c\") " pod="openshift-dns/dns-default-zgrjv" Jul 15 19:38:42.951484 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.951417 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86437276-af54-49be-acf6-91f82d55e92c-config-volume\") pod \"dns-default-zgrjv\" (UID: \"86437276-af54-49be-acf6-91f82d55e92c\") " pod="openshift-dns/dns-default-zgrjv" Jul 15 19:38:42.951484 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.951443 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-k8brr\" (UID: \"7a0eeb16-5201-43e8-a850-a7a3cdedc3a1\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k8brr" Jul 15 19:38:42.951484 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:42.951447 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:38:42.951706 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:42.951512 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls podName:86437276-af54-49be-acf6-91f82d55e92c nodeName:}" failed. No retries permitted until 2026-07-15 19:38:43.451492265 +0000 UTC m=+37.695998710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls") pod "dns-default-zgrjv" (UID: "86437276-af54-49be-acf6-91f82d55e92c") : secret "dns-default-metrics-tls" not found Jul 15 19:38:42.951706 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:42.951578 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 19:38:42.951706 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:42.951626 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert podName:7a0eeb16-5201-43e8-a850-a7a3cdedc3a1 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:43.451611621 +0000 UTC m=+37.696118074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-k8brr" (UID: "7a0eeb16-5201-43e8-a850-a7a3cdedc3a1") : secret "networking-console-plugin-cert" not found Jul 15 19:38:42.952050 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.951929 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86437276-af54-49be-acf6-91f82d55e92c-config-volume\") pod \"dns-default-zgrjv\" (UID: \"86437276-af54-49be-acf6-91f82d55e92c\") " pod="openshift-dns/dns-default-zgrjv" Jul 15 19:38:42.952050 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.951973 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-nginx-conf\") pod \"networking-console-plugin-6cf856758c-k8brr\" (UID: \"7a0eeb16-5201-43e8-a850-a7a3cdedc3a1\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k8brr" Jul 15 19:38:42.953575 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.953552 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d896f9b8-775c-4130-820c-59831ff12641-cert\") pod \"ingress-canary-2288d\" (UID: \"d896f9b8-775c-4130-820c-59831ff12641\") " pod="openshift-ingress-canary/ingress-canary-2288d" Jul 15 19:38:42.960652 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.960619 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5897\" (UniqueName: \"kubernetes.io/projected/d896f9b8-775c-4130-820c-59831ff12641-kube-api-access-g5897\") pod \"ingress-canary-2288d\" (UID: \"d896f9b8-775c-4130-820c-59831ff12641\") " pod="openshift-ingress-canary/ingress-canary-2288d" Jul 15 19:38:42.960828 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.960811 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8tlt\" (UniqueName: \"kubernetes.io/projected/86437276-af54-49be-acf6-91f82d55e92c-kube-api-access-l8tlt\") pod \"dns-default-zgrjv\" (UID: \"86437276-af54-49be-acf6-91f82d55e92c\") " pod="openshift-dns/dns-default-zgrjv" Jul 15 19:38:42.961292 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.961274 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpz6r\" (UniqueName: \"kubernetes.io/projected/49cd2b3f-c1fa-4cf8-a1b8-5b5583414a9b-kube-api-access-bpz6r\") pod \"network-check-source-869dcfcdcd-ffl9z\" (UID: \"49cd2b3f-c1fa-4cf8-a1b8-5b5583414a9b\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ffl9z" Jul 15 19:38:42.978581 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.978565 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-2ldzn" Jul 15 19:38:42.988408 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:42.988388 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" Jul 15 19:38:43.010519 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.010491 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftq5p" Jul 15 19:38:43.027117 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.027085 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-69x22" Jul 15 19:38:43.045357 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.045160 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zms8d" Jul 15 19:38:43.065187 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.064966 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ffl9z" Jul 15 19:38:43.073088 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.072902 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2288d" Jul 15 19:38:43.207823 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.203714 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-2ldzn"] Jul 15 19:38:43.233633 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.233076 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:38:43.233633 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.233283 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:38:43.233633 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.233506 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:38:43.236791 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.236732 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 19:38:43.237168 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.237148 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vqmsl\"" Jul 15 19:38:43.237412 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.237394 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lw97b\"" Jul 15 19:38:43.237655 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.237517 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 19:38:43.243702 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.243533 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-pmncb"] Jul 15 19:38:43.249432 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.249400 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftq5p"] Jul 15 19:38:43.250802 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:43.250774 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2691146_bc04_4c7b_bb0c_b3377762da6e.slice/crio-712bdc2c8c3fca5cca13ee0f8a40b47552c337f317a71fed06cf8ac0fad8926b WatchSource:0}: Error finding container 712bdc2c8c3fca5cca13ee0f8a40b47552c337f317a71fed06cf8ac0fad8926b: Status 404 returned error can't find the container with id 712bdc2c8c3fca5cca13ee0f8a40b47552c337f317a71fed06cf8ac0fad8926b Jul 15 19:38:43.266896 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:43.266868 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3227b7b_2925_413d_aaf3_aa5e5bb60a81.slice/crio-91b15326f575383ea27eeb7f4486743ae9407aa6460bfed771bc71573ad0d980 WatchSource:0}: Error finding container 91b15326f575383ea27eeb7f4486743ae9407aa6460bfed771bc71573ad0d980: Status 404 returned error can't find the container with id 91b15326f575383ea27eeb7f4486743ae9407aa6460bfed771bc71573ad0d980 Jul 15 19:38:43.281355 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.281249 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-69x22"] Jul 15 19:38:43.284810 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:43.284090 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae8e73f_008e_4977_8153_342f4a7460a5.slice/crio-f3846fc1911f410c312120e1f931ac36474ea0cffd3537906495a60277457d37 WatchSource:0}: Error finding container f3846fc1911f410c312120e1f931ac36474ea0cffd3537906495a60277457d37: Status 404 returned error can't find the container with id f3846fc1911f410c312120e1f931ac36474ea0cffd3537906495a60277457d37 Jul 15 19:38:43.285076 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.285055 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zms8d"] Jul 15 19:38:43.287981 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:43.287954 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99bcfa5f_a482_4e96_bc3f_a6e3365078d3.slice/crio-386b86421696ff7ca8f1b3ae0d4a607c4f396f002cc64e82decdb9e13d3e3813 WatchSource:0}: Error finding container 386b86421696ff7ca8f1b3ae0d4a607c4f396f002cc64e82decdb9e13d3e3813: Status 404 returned error can't find the container with id 386b86421696ff7ca8f1b3ae0d4a607c4f396f002cc64e82decdb9e13d3e3813 Jul 15 19:38:43.304907 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.304889 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-ffl9z"] Jul 15 19:38:43.306895 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:43.306874 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49cd2b3f_c1fa_4cf8_a1b8_5b5583414a9b.slice/crio-ef6d9254c5affedc1bc87179cd99bb2c894fcbb230b6f93e5507067305484138 WatchSource:0}: Error finding container ef6d9254c5affedc1bc87179cd99bb2c894fcbb230b6f93e5507067305484138: Status 404 returned error can't find the container with id ef6d9254c5affedc1bc87179cd99bb2c894fcbb230b6f93e5507067305484138 Jul 15 19:38:43.313056 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.313005 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2288d"] Jul 15 19:38:43.315058 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:43.315040 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd896f9b8_775c_4130_820c_59831ff12641.slice/crio-a26fbda8232b57886718cb1a791207a16ea4547f48279467072fb356b7a2e9ee WatchSource:0}: Error finding container a26fbda8232b57886718cb1a791207a16ea4547f48279467072fb356b7a2e9ee: Status 404 returned error can't find the container with id a26fbda8232b57886718cb1a791207a16ea4547f48279467072fb356b7a2e9ee Jul 15 19:38:43.355298 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.355210 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b90b66a6-7fd1-47c8-8250-83f56348498c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-957d7\" (UID: \"b90b66a6-7fd1-47c8-8250-83f56348498c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7" Jul 15 19:38:43.355298 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.355282 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:43.355495 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.355310 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-fksq2\" (UID: \"88a9eb45-3875-4d7f-aab1-e1aeb90333d6\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" Jul 15 19:38:43.355495 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.355326 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:43.355495 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.355348 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:43.355495 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:43.355414 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 15 19:38:43.355495 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:43.355431 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 15 19:38:43.355495 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:43.355435 2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 19:38:43.355495 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:43.355418 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 19:38:43.355495 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:43.355469 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle podName:eb9ebe84-37a6-4f73-880e-dad90c6453c7 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:44.355452672 +0000 UTC m=+38.599959114 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle") pod "router-default-656f8d6d5b-5lzhw" (UID: "eb9ebe84-37a6-4f73-880e-dad90c6453c7") : configmap references non-existent config key: service-ca.crt Jul 15 19:38:43.355495 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:43.355473 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d66974864-7mgdf: secret "image-registry-tls" not found Jul 15 19:38:43.355495 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:43.355495 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls podName:88a9eb45-3875-4d7f-aab1-e1aeb90333d6 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:44.355479376 +0000 UTC m=+38.599985815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-fksq2" (UID: "88a9eb45-3875-4d7f-aab1-e1aeb90333d6") : secret "cluster-monitoring-operator-tls" not found Jul 15 19:38:43.355858 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:43.355518 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b90b66a6-7fd1-47c8-8250-83f56348498c-samples-operator-tls podName:b90b66a6-7fd1-47c8-8250-83f56348498c nodeName:}" failed. No retries permitted until 2026-07-15 19:38:44.355509411 +0000 UTC m=+38.600015851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b90b66a6-7fd1-47c8-8250-83f56348498c-samples-operator-tls") pod "cluster-samples-operator-65db858894-957d7" (UID: "b90b66a6-7fd1-47c8-8250-83f56348498c") : secret "samples-operator-tls" not found Jul 15 19:38:43.355858 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:43.355530 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs podName:eb9ebe84-37a6-4f73-880e-dad90c6453c7 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:44.355523532 +0000 UTC m=+38.600029970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs") pod "router-default-656f8d6d5b-5lzhw" (UID: "eb9ebe84-37a6-4f73-880e-dad90c6453c7") : secret "router-metrics-certs-default" not found Jul 15 19:38:43.355858 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:43.355542 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls podName:72b9d425-51d4-4da9-8d59-bd071ec852bb nodeName:}" failed. No retries permitted until 2026-07-15 19:38:44.355536088 +0000 UTC m=+38.600042528 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls") pod "image-registry-6d66974864-7mgdf" (UID: "72b9d425-51d4-4da9-8d59-bd071ec852bb") : secret "image-registry-tls" not found Jul 15 19:38:43.384160 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.384111 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:43.408271 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.408241 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftq5p" event={"ID":"e3227b7b-2925-413d-aaf3-aa5e5bb60a81","Type":"ContainerStarted","Data":"91b15326f575383ea27eeb7f4486743ae9407aa6460bfed771bc71573ad0d980"} Jul 15 19:38:43.409218 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.409199 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zms8d" event={"ID":"99bcfa5f-a482-4e96-bc3f-a6e3365078d3","Type":"ContainerStarted","Data":"386b86421696ff7ca8f1b3ae0d4a607c4f396f002cc64e82decdb9e13d3e3813"} Jul 15 19:38:43.410171 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.410152 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2288d" event={"ID":"d896f9b8-775c-4130-820c-59831ff12641","Type":"ContainerStarted","Data":"a26fbda8232b57886718cb1a791207a16ea4547f48279467072fb356b7a2e9ee"} Jul 15 19:38:43.411180 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.411163 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-69x22" event={"ID":"2ae8e73f-008e-4977-8153-342f4a7460a5","Type":"ContainerStarted","Data":"f3846fc1911f410c312120e1f931ac36474ea0cffd3537906495a60277457d37"} Jul 15 19:38:43.412144 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.412127 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-2ldzn" event={"ID":"b21f8771-ffd5-4487-812b-bfbaa7718461","Type":"ContainerStarted","Data":"b15726fc5bfa387319e43539105222723669b480bfb242de1a6013e95bcab74c"} Jul 15 19:38:43.414738 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.414713 2554 generic.go:356] "Generic (PLEG): container finished" podID="c865faf5-fd82-4070-b169-a018a5c0bb0d" containerID="10cf96c24e9f7ff6d11a88e3576db3af270803b3de79506f22d60609f8126fcf" exitCode=0 Jul 15 19:38:43.414848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.414796 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9mp58" event={"ID":"c865faf5-fd82-4070-b169-a018a5c0bb0d","Type":"ContainerDied","Data":"10cf96c24e9f7ff6d11a88e3576db3af270803b3de79506f22d60609f8126fcf"} Jul 15 19:38:43.415901 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.415871 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ffl9z" event={"ID":"49cd2b3f-c1fa-4cf8-a1b8-5b5583414a9b","Type":"ContainerStarted","Data":"ef6d9254c5affedc1bc87179cd99bb2c894fcbb230b6f93e5507067305484138"} Jul 15 19:38:43.416981 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.416959 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" event={"ID":"e2691146-bc04-4c7b-bb0c-b3377762da6e","Type":"ContainerStarted","Data":"712bdc2c8c3fca5cca13ee0f8a40b47552c337f317a71fed06cf8ac0fad8926b"} Jul 15 19:38:43.456288 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.456104 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls\") pod \"dns-default-zgrjv\" (UID: \"86437276-af54-49be-acf6-91f82d55e92c\") " pod="openshift-dns/dns-default-zgrjv" Jul 15 19:38:43.456462 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.456333 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-k8brr\" (UID: \"7a0eeb16-5201-43e8-a850-a7a3cdedc3a1\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k8brr" Jul 15 19:38:43.456462 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:43.456256 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:38:43.456462 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:43.456409 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls podName:86437276-af54-49be-acf6-91f82d55e92c nodeName:}" failed. No retries permitted until 2026-07-15 19:38:44.456392259 +0000 UTC m=+38.700898697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls") pod "dns-default-zgrjv" (UID: "86437276-af54-49be-acf6-91f82d55e92c") : secret "dns-default-metrics-tls" not found Jul 15 19:38:43.456462 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:43.456440 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 19:38:43.456613 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:43.456488 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert podName:7a0eeb16-5201-43e8-a850-a7a3cdedc3a1 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:44.456473853 +0000 UTC m=+38.700980299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-k8brr" (UID: "7a0eeb16-5201-43e8-a850-a7a3cdedc3a1") : secret "networking-console-plugin-cert" not found Jul 15 19:38:43.478745 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.478673 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:43.582190 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.582139 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:43.649128 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.649051 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:43.912233 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.911958 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:43.982246 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:43.982066 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:44.366011 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:44.365483 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b90b66a6-7fd1-47c8-8250-83f56348498c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-957d7\" (UID: \"b90b66a6-7fd1-47c8-8250-83f56348498c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7" Jul 15 19:38:44.366011 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:44.365671 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:44.366011 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:44.365679 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 15 19:38:44.366011 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:44.365724 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-fksq2\" (UID: \"88a9eb45-3875-4d7f-aab1-e1aeb90333d6\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" Jul 15 19:38:44.366011 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:44.365744 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b90b66a6-7fd1-47c8-8250-83f56348498c-samples-operator-tls podName:b90b66a6-7fd1-47c8-8250-83f56348498c nodeName:}" failed. No retries permitted until 2026-07-15 19:38:46.365725939 +0000 UTC m=+40.610232397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b90b66a6-7fd1-47c8-8250-83f56348498c-samples-operator-tls") pod "cluster-samples-operator-65db858894-957d7" (UID: "b90b66a6-7fd1-47c8-8250-83f56348498c") : secret "samples-operator-tls" not found Jul 15 19:38:44.366011 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:44.365793 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:44.366011 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:44.365822 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 15 19:38:44.366011 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:44.365830 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:44.366011 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:44.365862 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls podName:88a9eb45-3875-4d7f-aab1-e1aeb90333d6 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:46.365849737 +0000 UTC m=+40.610356179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-fksq2" (UID: "88a9eb45-3875-4d7f-aab1-e1aeb90333d6") : secret "cluster-monitoring-operator-tls" not found Jul 15 19:38:44.366011 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:44.365865 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 19:38:44.366011 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:44.365887 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d66974864-7mgdf: secret "image-registry-tls" not found Jul 15 19:38:44.366011 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:44.365892 2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 19:38:44.366011 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:44.365921 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs podName:eb9ebe84-37a6-4f73-880e-dad90c6453c7 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:46.365911407 +0000 UTC m=+40.610417848 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs") pod "router-default-656f8d6d5b-5lzhw" (UID: "eb9ebe84-37a6-4f73-880e-dad90c6453c7") : secret "router-metrics-certs-default" not found Jul 15 19:38:44.366011 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:44.365936 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls podName:72b9d425-51d4-4da9-8d59-bd071ec852bb nodeName:}" failed. No retries permitted until 2026-07-15 19:38:46.365929106 +0000 UTC m=+40.610435545 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls") pod "image-registry-6d66974864-7mgdf" (UID: "72b9d425-51d4-4da9-8d59-bd071ec852bb") : secret "image-registry-tls" not found Jul 15 19:38:44.366011 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:44.366006 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle podName:eb9ebe84-37a6-4f73-880e-dad90c6453c7 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:46.365988681 +0000 UTC m=+40.610495136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle") pod "router-default-656f8d6d5b-5lzhw" (UID: "eb9ebe84-37a6-4f73-880e-dad90c6453c7") : configmap references non-existent config key: service-ca.crt Jul 15 19:38:44.428786 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:44.427977 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zms8d" event={"ID":"99bcfa5f-a482-4e96-bc3f-a6e3365078d3","Type":"ContainerStarted","Data":"7d9f14f6fcd50c612fe482757f3eb32a9b69b135deb35b884983f9d0203f3a63"} Jul 15 19:38:44.434689 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:44.434656 2554 generic.go:356] "Generic (PLEG): container finished" podID="c865faf5-fd82-4070-b169-a018a5c0bb0d" containerID="9ce73185c74d0f001504ea351c4bea764dfbc0d86fde9ab848e98e8ca02fbb4d" exitCode=0 Jul 15 19:38:44.434863 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:44.434728 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9mp58" event={"ID":"c865faf5-fd82-4070-b169-a018a5c0bb0d","Type":"ContainerDied","Data":"9ce73185c74d0f001504ea351c4bea764dfbc0d86fde9ab848e98e8ca02fbb4d"} Jul 15 19:38:44.470184 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:44.469301 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls\") pod \"dns-default-zgrjv\" (UID: \"86437276-af54-49be-acf6-91f82d55e92c\") " pod="openshift-dns/dns-default-zgrjv" Jul 15 19:38:44.470184 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:44.469387 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-k8brr\" (UID: \"7a0eeb16-5201-43e8-a850-a7a3cdedc3a1\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k8brr" Jul 15 19:38:44.470184 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:44.469637 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 19:38:44.470184 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:44.469701 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert podName:7a0eeb16-5201-43e8-a850-a7a3cdedc3a1 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:46.469680873 +0000 UTC m=+40.714187316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-k8brr" (UID: "7a0eeb16-5201-43e8-a850-a7a3cdedc3a1") : secret "networking-console-plugin-cert" not found Jul 15 19:38:44.470184 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:44.470025 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:38:44.470184 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:44.470070 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls podName:86437276-af54-49be-acf6-91f82d55e92c nodeName:}" failed. No retries permitted until 2026-07-15 19:38:46.470056835 +0000 UTC m=+40.714563278 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls") pod "dns-default-zgrjv" (UID: "86437276-af54-49be-acf6-91f82d55e92c") : secret "dns-default-metrics-tls" not found Jul 15 19:38:44.481231 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:44.481146 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zms8d" podStartSLOduration=19.477288747 podStartE2EDuration="20.481126148s" totalImagesPullingTime="1.003837401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:44.449433001 +0000 UTC m=+38.693939465" watchObservedRunningTime="2026-07-15 19:38:44.481126148 +0000 UTC m=+38.725632611" Jul 15 19:38:44.550479 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:44.550451 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:44.650935 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:44.650806 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:45.445767 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:45.445677 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9mp58" event={"ID":"c865faf5-fd82-4070-b169-a018a5c0bb0d","Type":"ContainerStarted","Data":"7f08f72a3311e9737a720d28b4a8d5757dee1df8377b1581f79bd27020e18d69"} Jul 15 19:38:45.473254 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:45.472323 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9mp58" podStartSLOduration=9.305477808 podStartE2EDuration="39.472302509s" totalImagesPullingTime="30.166824701s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:38:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:45.470430701 +0000 UTC m=+39.714937163" watchObservedRunningTime="2026-07-15 19:38:45.472302509 +0000 UTC m=+39.716808971" Jul 15 19:38:45.597155 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:45.596900 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:45.657214 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:45.656989 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:46.388660 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:46.388617 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b90b66a6-7fd1-47c8-8250-83f56348498c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-957d7\" (UID: \"b90b66a6-7fd1-47c8-8250-83f56348498c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7" Jul 15 19:38:46.388895 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:46.388735 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:46.388895 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:46.388800 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-fksq2\" (UID: \"88a9eb45-3875-4d7f-aab1-e1aeb90333d6\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" Jul 15 19:38:46.388895 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:46.388832 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:46.388895 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:46.388867 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:46.388895 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:46.388805 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 15 19:38:46.389157 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:46.388922 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 19:38:46.389157 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:46.388939 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b90b66a6-7fd1-47c8-8250-83f56348498c-samples-operator-tls podName:b90b66a6-7fd1-47c8-8250-83f56348498c nodeName:}" failed. No retries permitted until 2026-07-15 19:38:50.388919962 +0000 UTC m=+44.633426401 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b90b66a6-7fd1-47c8-8250-83f56348498c-samples-operator-tls") pod "cluster-samples-operator-65db858894-957d7" (UID: "b90b66a6-7fd1-47c8-8250-83f56348498c") : secret "samples-operator-tls" not found Jul 15 19:38:46.389157 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:46.388943 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d66974864-7mgdf: secret "image-registry-tls" not found Jul 15 19:38:46.389157 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:46.388967 2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 19:38:46.389157 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:46.388868 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 15 19:38:46.389157 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:46.389001 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls podName:72b9d425-51d4-4da9-8d59-bd071ec852bb nodeName:}" failed. No retries permitted until 2026-07-15 19:38:50.388981305 +0000 UTC m=+44.633487750 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls") pod "image-registry-6d66974864-7mgdf" (UID: "72b9d425-51d4-4da9-8d59-bd071ec852bb") : secret "image-registry-tls" not found Jul 15 19:38:46.389157 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:46.389020 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle podName:eb9ebe84-37a6-4f73-880e-dad90c6453c7 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:50.389010305 +0000 UTC m=+44.633516748 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle") pod "router-default-656f8d6d5b-5lzhw" (UID: "eb9ebe84-37a6-4f73-880e-dad90c6453c7") : configmap references non-existent config key: service-ca.crt Jul 15 19:38:46.389157 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:46.389043 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls podName:88a9eb45-3875-4d7f-aab1-e1aeb90333d6 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:50.3890265 +0000 UTC m=+44.633532953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-fksq2" (UID: "88a9eb45-3875-4d7f-aab1-e1aeb90333d6") : secret "cluster-monitoring-operator-tls" not found Jul 15 19:38:46.389157 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:46.389063 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs podName:eb9ebe84-37a6-4f73-880e-dad90c6453c7 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:50.389052403 +0000 UTC m=+44.633558860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs") pod "router-default-656f8d6d5b-5lzhw" (UID: "eb9ebe84-37a6-4f73-880e-dad90c6453c7") : secret "router-metrics-certs-default" not found Jul 15 19:38:46.490295 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:46.490257 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-k8brr\" (UID: \"7a0eeb16-5201-43e8-a850-a7a3cdedc3a1\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k8brr" Jul 15 19:38:46.490532 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:46.490514 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls\") pod \"dns-default-zgrjv\" (UID: \"86437276-af54-49be-acf6-91f82d55e92c\") " pod="openshift-dns/dns-default-zgrjv" Jul 15 19:38:46.490710 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:46.490686 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:38:46.490858 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:46.490775 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls podName:86437276-af54-49be-acf6-91f82d55e92c nodeName:}" failed. No retries permitted until 2026-07-15 19:38:50.490738085 +0000 UTC m=+44.735244532 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls") pod "dns-default-zgrjv" (UID: "86437276-af54-49be-acf6-91f82d55e92c") : secret "dns-default-metrics-tls" not found Jul 15 19:38:46.491176 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:46.491154 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 19:38:46.491284 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:46.491220 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert podName:7a0eeb16-5201-43e8-a850-a7a3cdedc3a1 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:50.491203693 +0000 UTC m=+44.735710147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-k8brr" (UID: "7a0eeb16-5201-43e8-a850-a7a3cdedc3a1") : secret "networking-console-plugin-cert" not found Jul 15 19:38:46.894880 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:46.894843 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:46.949171 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:46.949132 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:50.403650 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:50.403572 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:50.429723 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:50.429691 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:50.429915 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:50.429738 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-fksq2\" (UID: \"88a9eb45-3875-4d7f-aab1-e1aeb90333d6\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" Jul 15 19:38:50.429915 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:50.429783 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:50.429915 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:50.429821 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:50.429915 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:50.429854 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 19:38:50.429915 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:50.429873 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d66974864-7mgdf: secret "image-registry-tls" not found Jul 15 19:38:50.429915 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:50.429891 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b90b66a6-7fd1-47c8-8250-83f56348498c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-957d7\" (UID: \"b90b66a6-7fd1-47c8-8250-83f56348498c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7" Jul 15 19:38:50.429915 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:50.429927 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls podName:72b9d425-51d4-4da9-8d59-bd071ec852bb nodeName:}" failed. No retries permitted until 2026-07-15 19:38:58.429907832 +0000 UTC m=+52.674414272 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls") pod "image-registry-6d66974864-7mgdf" (UID: "72b9d425-51d4-4da9-8d59-bd071ec852bb") : secret "image-registry-tls" not found Jul 15 19:38:50.430228 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:50.429928 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 15 19:38:50.430228 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:50.429948 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle podName:eb9ebe84-37a6-4f73-880e-dad90c6453c7 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:58.429933509 +0000 UTC m=+52.674439967 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle") pod "router-default-656f8d6d5b-5lzhw" (UID: "eb9ebe84-37a6-4f73-880e-dad90c6453c7") : configmap references non-existent config key: service-ca.crt Jul 15 19:38:50.430228 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:50.429985 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls podName:88a9eb45-3875-4d7f-aab1-e1aeb90333d6 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:58.42996903 +0000 UTC m=+52.674475469 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-fksq2" (UID: "88a9eb45-3875-4d7f-aab1-e1aeb90333d6") : secret "cluster-monitoring-operator-tls" not found Jul 15 19:38:50.430228 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:50.429984 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 15 19:38:50.430228 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:50.430076 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b90b66a6-7fd1-47c8-8250-83f56348498c-samples-operator-tls podName:b90b66a6-7fd1-47c8-8250-83f56348498c nodeName:}" failed. No retries permitted until 2026-07-15 19:38:58.430068049 +0000 UTC m=+52.674574492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b90b66a6-7fd1-47c8-8250-83f56348498c-samples-operator-tls") pod "cluster-samples-operator-65db858894-957d7" (UID: "b90b66a6-7fd1-47c8-8250-83f56348498c") : secret "samples-operator-tls" not found Jul 15 19:38:50.430228 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:50.429990 2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 19:38:50.430228 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:50.430107 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs podName:eb9ebe84-37a6-4f73-880e-dad90c6453c7 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:58.430101549 +0000 UTC m=+52.674607990 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs") pod "router-default-656f8d6d5b-5lzhw" (UID: "eb9ebe84-37a6-4f73-880e-dad90c6453c7") : secret "router-metrics-certs-default" not found Jul 15 19:38:50.510482 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:50.510433 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:50.530772 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:50.530724 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-k8brr\" (UID: \"7a0eeb16-5201-43e8-a850-a7a3cdedc3a1\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k8brr" Jul 15 19:38:50.530889 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:50.530858 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls\") pod \"dns-default-zgrjv\" (UID: \"86437276-af54-49be-acf6-91f82d55e92c\") " pod="openshift-dns/dns-default-zgrjv" Jul 15 19:38:50.530889 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:50.530868 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 19:38:50.531008 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:50.530922 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert podName:7a0eeb16-5201-43e8-a850-a7a3cdedc3a1 nodeName:}" failed. No retries permitted until 2026-07-15 19:38:58.530908338 +0000 UTC m=+52.775414780 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-k8brr" (UID: "7a0eeb16-5201-43e8-a850-a7a3cdedc3a1") : secret "networking-console-plugin-cert" not found Jul 15 19:38:50.531008 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:50.530971 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:38:50.531121 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:50.531018 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls podName:86437276-af54-49be-acf6-91f82d55e92c nodeName:}" failed. No retries permitted until 2026-07-15 19:38:58.53100832 +0000 UTC m=+52.775514759 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls") pod "dns-default-zgrjv" (UID: "86437276-af54-49be-acf6-91f82d55e92c") : secret "dns-default-metrics-tls" not found Jul 15 19:38:51.463277 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:51.463235 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2288d" event={"ID":"d896f9b8-775c-4130-820c-59831ff12641","Type":"ContainerStarted","Data":"cb5c1a16a2196053cab035751edfd4d7fd45191f388b9605b7abe195a4fac687"} Jul 15 19:38:51.465179 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:51.465146 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-69x22" event={"ID":"2ae8e73f-008e-4977-8153-342f4a7460a5","Type":"ContainerStarted","Data":"054c270ef7c29f645e8d1dc75bec1ab247e8b833d5d5dd8f4c0cf9ea8ea682b0"} Jul 15 19:38:51.466995 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:51.466973 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/0.log" Jul 15 19:38:51.467102 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:51.467014 2554 generic.go:356] "Generic (PLEG): container finished" podID="b21f8771-ffd5-4487-812b-bfbaa7718461" containerID="810721bb4a4e3b67748f7382b23a26f97589bfd496e9ecbe0500d009effcade4" exitCode=255 Jul 15 19:38:51.467102 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:51.467093 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-2ldzn" event={"ID":"b21f8771-ffd5-4487-812b-bfbaa7718461","Type":"ContainerDied","Data":"810721bb4a4e3b67748f7382b23a26f97589bfd496e9ecbe0500d009effcade4"} Jul 15 19:38:51.467494 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:51.467477 2554 scope.go:117] "RemoveContainer" containerID="810721bb4a4e3b67748f7382b23a26f97589bfd496e9ecbe0500d009effcade4" Jul 15 19:38:51.470689 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:51.470459 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ffl9z" event={"ID":"49cd2b3f-c1fa-4cf8-a1b8-5b5583414a9b","Type":"ContainerStarted","Data":"8d43673ac1f664bc80738a44ae6235c251e66bb324047237f0dd37f2ae16b1ef"} Jul 15 19:38:51.472134 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:51.472110 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" event={"ID":"e2691146-bc04-4c7b-bb0c-b3377762da6e","Type":"ContainerStarted","Data":"790482c7093bcc9c0a065aee3aa1e0445c3ae2561feebf7b42d91e0bd12896dd"} Jul 15 19:38:51.473815 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:51.473790 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftq5p" event={"ID":"e3227b7b-2925-413d-aaf3-aa5e5bb60a81","Type":"ContainerStarted","Data":"068c68a4c5d0c989441d12f3fa3d9e35071434763040d1b7853c3a0d83627115"} Jul 15 19:38:51.481804 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:51.481712 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2288d" podStartSLOduration=2.295692837 podStartE2EDuration="9.481694405s" totalImagesPullingTime="7.186001568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:51.480026261 +0000 UTC m=+45.724532722" watchObservedRunningTime="2026-07-15 19:38:51.481694405 +0000 UTC m=+45.726200870" Jul 15 19:38:51.515567 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:51.515513 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" podStartSLOduration=20.302623631 podStartE2EDuration="27.515489525s" totalImagesPullingTime="7.212865894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:51.498306981 +0000 UTC m=+45.742813443" watchObservedRunningTime="2026-07-15 19:38:51.515489525 +0000 UTC m=+45.759995987" Jul 15 19:38:51.516938 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:51.516894 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftq5p" podStartSLOduration=19.284329083 podStartE2EDuration="26.516877904s" totalImagesPullingTime="7.232548821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:51.515962163 +0000 UTC m=+45.760468618" watchObservedRunningTime="2026-07-15 19:38:51.516877904 +0000 UTC m=+45.761384366" Jul 15 19:38:51.537526 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:51.537318 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-69x22" podStartSLOduration=19.321019822 podStartE2EDuration="26.537299598s" totalImagesPullingTime="7.216279776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:51.536548134 +0000 UTC m=+45.781054596" watchObservedRunningTime="2026-07-15 19:38:51.537299598 +0000 UTC m=+45.781806058" Jul 15 19:38:51.559013 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:51.558799 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ffl9z" podStartSLOduration=19.354340096 podStartE2EDuration="26.558778018s" totalImagesPullingTime="7.204437922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:51.557605496 +0000 UTC m=+45.802111957" watchObservedRunningTime="2026-07-15 19:38:51.558778018 +0000 UTC m=+45.803284482" Jul 15 19:38:51.929608 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:51.929567 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:52.425603 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.425572 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-87gx7"] Jul 15 19:38:52.455719 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.455687 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-87gx7"] Jul 15 19:38:52.455902 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.455838 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-87gx7" Jul 15 19:38:52.459267 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.459242 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 15 19:38:52.459953 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.459929 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 15 19:38:52.460054 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.459972 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wmt48\"" Jul 15 19:38:52.479414 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.479391 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 19:38:52.479850 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.479834 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/0.log" Jul 15 19:38:52.479922 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.479873 2554 generic.go:356] "Generic (PLEG): container finished" podID="b21f8771-ffd5-4487-812b-bfbaa7718461" containerID="ce6d05724b8d9ad112e9ce5a9d7db0f3771fb5536965b5bb11a1308554214fcd" exitCode=255 Jul 15 19:38:52.479970 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.479940 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-2ldzn" event={"ID":"b21f8771-ffd5-4487-812b-bfbaa7718461","Type":"ContainerDied","Data":"ce6d05724b8d9ad112e9ce5a9d7db0f3771fb5536965b5bb11a1308554214fcd"} Jul 15 19:38:52.480034 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.480002 2554 scope.go:117] "RemoveContainer" containerID="810721bb4a4e3b67748f7382b23a26f97589bfd496e9ecbe0500d009effcade4" Jul 15 19:38:52.480231 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.480206 2554 scope.go:117] "RemoveContainer" containerID="ce6d05724b8d9ad112e9ce5a9d7db0f3771fb5536965b5bb11a1308554214fcd" Jul 15 19:38:52.480415 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:52.480392 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-2ldzn_openshift-console-operator(b21f8771-ffd5-4487-812b-bfbaa7718461)\"" pod="openshift-console-operator/console-operator-7866497494-2ldzn" podUID="b21f8771-ffd5-4487-812b-bfbaa7718461" Jul 15 19:38:52.552095 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.551746 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4rts\" (UniqueName: \"kubernetes.io/projected/bd5cd4f0-d31d-4f1f-877d-53537b0fc122-kube-api-access-r4rts\") pod \"migrator-7f59b54f6c-87gx7\" (UID: \"bd5cd4f0-d31d-4f1f-877d-53537b0fc122\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-87gx7" Jul 15 19:38:52.653355 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.653319 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4rts\" (UniqueName: \"kubernetes.io/projected/bd5cd4f0-d31d-4f1f-877d-53537b0fc122-kube-api-access-r4rts\") pod \"migrator-7f59b54f6c-87gx7\" (UID: \"bd5cd4f0-d31d-4f1f-877d-53537b0fc122\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-87gx7" Jul 15 19:38:52.664230 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.664199 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4rts\" (UniqueName: \"kubernetes.io/projected/bd5cd4f0-d31d-4f1f-877d-53537b0fc122-kube-api-access-r4rts\") pod \"migrator-7f59b54f6c-87gx7\" (UID: \"bd5cd4f0-d31d-4f1f-877d-53537b0fc122\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-87gx7" Jul 15 19:38:52.679102 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.679023 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:52.752921 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.752885 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:52.764869 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.764844 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-87gx7" Jul 15 19:38:52.840148 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.840116 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5rb4w"] Jul 15 19:38:52.870617 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.870135 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:52.870617 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.870200 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5rb4w"] Jul 15 19:38:52.873141 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.873050 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 19:38:52.874205 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.874077 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hjcsl\"" Jul 15 19:38:52.874205 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.874081 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 19:38:52.893341 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.893207 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:52.956865 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.956781 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38c84af0-f6d3-4217-ba8a-87fb6912c97e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:52.957004 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.956910 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38c84af0-f6d3-4217-ba8a-87fb6912c97e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:52.957004 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.956952 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/38c84af0-f6d3-4217-ba8a-87fb6912c97e-crio-socket\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:52.957004 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.956991 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/38c84af0-f6d3-4217-ba8a-87fb6912c97e-data-volume\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:52.957102 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.957048 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbr85\" (UniqueName: \"kubernetes.io/projected/38c84af0-f6d3-4217-ba8a-87fb6912c97e-kube-api-access-xbr85\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:52.978816 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.978789 2554 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-2ldzn" Jul 15 19:38:52.978951 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.978837 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-2ldzn" Jul 15 19:38:52.980666 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:52.980613 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-87gx7"] Jul 15 19:38:52.984350 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:52.984323 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd5cd4f0_d31d_4f1f_877d_53537b0fc122.slice/crio-5b3ad278f37f4a487f6efd5998b503840ab2deab1590cf1b9da664fc57a0fcc5 WatchSource:0}: Error finding container 5b3ad278f37f4a487f6efd5998b503840ab2deab1590cf1b9da664fc57a0fcc5: Status 404 returned error can't find the container with id 5b3ad278f37f4a487f6efd5998b503840ab2deab1590cf1b9da664fc57a0fcc5 Jul 15 19:38:53.012073 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:53.012034 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:53.057959 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:53.057929 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38c84af0-f6d3-4217-ba8a-87fb6912c97e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:53.058103 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:53.057975 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/38c84af0-f6d3-4217-ba8a-87fb6912c97e-crio-socket\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:53.058103 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:53.058014 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/38c84af0-f6d3-4217-ba8a-87fb6912c97e-data-volume\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:53.058103 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:53.058057 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:38:53.058247 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:53.058127 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/38c84af0-f6d3-4217-ba8a-87fb6912c97e-kube-rbac-proxy-cm podName:38c84af0-f6d3-4217-ba8a-87fb6912c97e nodeName:}" failed. No retries permitted until 2026-07-15 19:38:53.558110507 +0000 UTC m=+47.802616946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/38c84af0-f6d3-4217-ba8a-87fb6912c97e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5rb4w" (UID: "38c84af0-f6d3-4217-ba8a-87fb6912c97e") : configmap "kube-rbac-proxy" not found Jul 15 19:38:53.058247 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:53.058186 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/38c84af0-f6d3-4217-ba8a-87fb6912c97e-crio-socket\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:53.058247 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:53.058188 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbr85\" (UniqueName: \"kubernetes.io/projected/38c84af0-f6d3-4217-ba8a-87fb6912c97e-kube-api-access-xbr85\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:53.058376 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:53.058253 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38c84af0-f6d3-4217-ba8a-87fb6912c97e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:53.058376 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:53.058291 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/38c84af0-f6d3-4217-ba8a-87fb6912c97e-data-volume\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:53.058525 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:53.058414 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:38:53.058525 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:53.058480 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38c84af0-f6d3-4217-ba8a-87fb6912c97e-insights-runtime-extractor-tls podName:38c84af0-f6d3-4217-ba8a-87fb6912c97e nodeName:}" failed. No retries permitted until 2026-07-15 19:38:53.558461007 +0000 UTC m=+47.802967472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38c84af0-f6d3-4217-ba8a-87fb6912c97e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5rb4w" (UID: "38c84af0-f6d3-4217-ba8a-87fb6912c97e") : secret "insights-runtime-extractor-tls" not found Jul 15 19:38:53.067266 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:53.067241 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbr85\" (UniqueName: \"kubernetes.io/projected/38c84af0-f6d3-4217-ba8a-87fb6912c97e-kube-api-access-xbr85\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:53.115822 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:53.115791 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:53.485051 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:53.485022 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 19:38:53.485477 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:53.485390 2554 scope.go:117] "RemoveContainer" containerID="ce6d05724b8d9ad112e9ce5a9d7db0f3771fb5536965b5bb11a1308554214fcd" Jul 15 19:38:53.485571 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:53.485548 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-2ldzn_openshift-console-operator(b21f8771-ffd5-4487-812b-bfbaa7718461)\"" pod="openshift-console-operator/console-operator-7866497494-2ldzn" podUID="b21f8771-ffd5-4487-812b-bfbaa7718461" Jul 15 19:38:53.486238 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:53.486217 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-87gx7" event={"ID":"bd5cd4f0-d31d-4f1f-877d-53537b0fc122","Type":"ContainerStarted","Data":"5b3ad278f37f4a487f6efd5998b503840ab2deab1590cf1b9da664fc57a0fcc5"} Jul 15 19:38:53.562951 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:53.562917 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38c84af0-f6d3-4217-ba8a-87fb6912c97e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:53.563126 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:53.563089 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38c84af0-f6d3-4217-ba8a-87fb6912c97e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:53.563405 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:53.563378 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:38:53.563532 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:53.563435 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:38:53.563532 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:53.563445 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/38c84af0-f6d3-4217-ba8a-87fb6912c97e-kube-rbac-proxy-cm podName:38c84af0-f6d3-4217-ba8a-87fb6912c97e nodeName:}" failed. No retries permitted until 2026-07-15 19:38:54.563428074 +0000 UTC m=+48.807934514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/38c84af0-f6d3-4217-ba8a-87fb6912c97e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5rb4w" (UID: "38c84af0-f6d3-4217-ba8a-87fb6912c97e") : configmap "kube-rbac-proxy" not found Jul 15 19:38:53.563532 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:53.563491 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38c84af0-f6d3-4217-ba8a-87fb6912c97e-insights-runtime-extractor-tls podName:38c84af0-f6d3-4217-ba8a-87fb6912c97e nodeName:}" failed. No retries permitted until 2026-07-15 19:38:54.563478695 +0000 UTC m=+48.807985133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38c84af0-f6d3-4217-ba8a-87fb6912c97e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5rb4w" (UID: "38c84af0-f6d3-4217-ba8a-87fb6912c97e") : secret "insights-runtime-extractor-tls" not found Jul 15 19:38:54.489280 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.489256 2554 scope.go:117] "RemoveContainer" containerID="ce6d05724b8d9ad112e9ce5a9d7db0f3771fb5536965b5bb11a1308554214fcd" Jul 15 19:38:54.489642 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:54.489419 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-2ldzn_openshift-console-operator(b21f8771-ffd5-4487-812b-bfbaa7718461)\"" pod="openshift-console-operator/console-operator-7866497494-2ldzn" podUID="b21f8771-ffd5-4487-812b-bfbaa7718461" Jul 15 19:38:54.558817 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.558783 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-ldd9p"] Jul 15 19:38:54.573261 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.573232 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38c84af0-f6d3-4217-ba8a-87fb6912c97e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:54.573432 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.573376 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38c84af0-f6d3-4217-ba8a-87fb6912c97e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:54.573432 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:54.573383 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:38:54.573524 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:54.573441 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38c84af0-f6d3-4217-ba8a-87fb6912c97e-insights-runtime-extractor-tls podName:38c84af0-f6d3-4217-ba8a-87fb6912c97e nodeName:}" failed. No retries permitted until 2026-07-15 19:38:56.573424307 +0000 UTC m=+50.817930761 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38c84af0-f6d3-4217-ba8a-87fb6912c97e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5rb4w" (UID: "38c84af0-f6d3-4217-ba8a-87fb6912c97e") : secret "insights-runtime-extractor-tls" not found Jul 15 19:38:54.573524 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:54.573458 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:38:54.573524 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:54.573514 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/38c84af0-f6d3-4217-ba8a-87fb6912c97e-kube-rbac-proxy-cm podName:38c84af0-f6d3-4217-ba8a-87fb6912c97e nodeName:}" failed. No retries permitted until 2026-07-15 19:38:56.573496216 +0000 UTC m=+50.818002664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/38c84af0-f6d3-4217-ba8a-87fb6912c97e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5rb4w" (UID: "38c84af0-f6d3-4217-ba8a-87fb6912c97e") : configmap "kube-rbac-proxy" not found Jul 15 19:38:54.577730 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.577710 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-ldd9p"] Jul 15 19:38:54.577836 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.577809 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-ldd9p" Jul 15 19:38:54.580929 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.580910 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 15 19:38:54.581060 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.580989 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 15 19:38:54.582246 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.582219 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 15 19:38:54.582335 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.582254 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 15 19:38:54.582335 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.582222 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hnlj8\"" Jul 15 19:38:54.592281 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.592254 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:54.674133 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.674098 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwvks\" (UniqueName: \"kubernetes.io/projected/2edbfdc0-a0ec-4e17-897a-4e44954b19a9-kube-api-access-nwvks\") pod \"service-ca-79f68b9f4f-ldd9p\" (UID: \"2edbfdc0-a0ec-4e17-897a-4e44954b19a9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ldd9p" Jul 15 19:38:54.674281 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.674142 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2edbfdc0-a0ec-4e17-897a-4e44954b19a9-signing-cabundle\") pod \"service-ca-79f68b9f4f-ldd9p\" (UID: \"2edbfdc0-a0ec-4e17-897a-4e44954b19a9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ldd9p" Jul 15 19:38:54.674281 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.674259 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2edbfdc0-a0ec-4e17-897a-4e44954b19a9-signing-key\") pod \"service-ca-79f68b9f4f-ldd9p\" (UID: \"2edbfdc0-a0ec-4e17-897a-4e44954b19a9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ldd9p" Jul 15 19:38:54.694069 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.694036 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:54.774906 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.774878 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2edbfdc0-a0ec-4e17-897a-4e44954b19a9-signing-key\") pod \"service-ca-79f68b9f4f-ldd9p\" (UID: \"2edbfdc0-a0ec-4e17-897a-4e44954b19a9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ldd9p" Jul 15 19:38:54.775030 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.774999 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwvks\" (UniqueName: \"kubernetes.io/projected/2edbfdc0-a0ec-4e17-897a-4e44954b19a9-kube-api-access-nwvks\") pod \"service-ca-79f68b9f4f-ldd9p\" (UID: \"2edbfdc0-a0ec-4e17-897a-4e44954b19a9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ldd9p" Jul 15 19:38:54.775111 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.775035 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2edbfdc0-a0ec-4e17-897a-4e44954b19a9-signing-cabundle\") pod \"service-ca-79f68b9f4f-ldd9p\" (UID: \"2edbfdc0-a0ec-4e17-897a-4e44954b19a9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ldd9p" Jul 15 19:38:54.775607 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.775587 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2edbfdc0-a0ec-4e17-897a-4e44954b19a9-signing-cabundle\") pod \"service-ca-79f68b9f4f-ldd9p\" (UID: \"2edbfdc0-a0ec-4e17-897a-4e44954b19a9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ldd9p" Jul 15 19:38:54.777325 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.777303 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2edbfdc0-a0ec-4e17-897a-4e44954b19a9-signing-key\") pod \"service-ca-79f68b9f4f-ldd9p\" (UID: \"2edbfdc0-a0ec-4e17-897a-4e44954b19a9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ldd9p" Jul 15 19:38:54.784599 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.784578 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwvks\" (UniqueName: \"kubernetes.io/projected/2edbfdc0-a0ec-4e17-897a-4e44954b19a9-kube-api-access-nwvks\") pod \"service-ca-79f68b9f4f-ldd9p\" (UID: \"2edbfdc0-a0ec-4e17-897a-4e44954b19a9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ldd9p" Jul 15 19:38:54.887050 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:54.887025 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-ldd9p" Jul 15 19:38:55.000646 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:55.000565 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:55.047290 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:55.047268 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-ldd9p"] Jul 15 19:38:55.049689 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:38:55.049662 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2edbfdc0_a0ec_4e17_897a_4e44954b19a9.slice/crio-6663ff1e3f556facb6ae192a4d429ca0e6efd0ffb2f502f2397882971374d96b WatchSource:0}: Error finding container 6663ff1e3f556facb6ae192a4d429ca0e6efd0ffb2f502f2397882971374d96b: Status 404 returned error can't find the container with id 6663ff1e3f556facb6ae192a4d429ca0e6efd0ffb2f502f2397882971374d96b Jul 15 19:38:55.051695 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:55.051633 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zswzk_67b2bcf5-c147-4a76-b620-b0cec94a0132/dns-node-resolver/0.log" Jul 15 19:38:55.102085 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:55.102054 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:55.494246 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:55.494198 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-ldd9p" event={"ID":"2edbfdc0-a0ec-4e17-897a-4e44954b19a9","Type":"ContainerStarted","Data":"affc349156001fdb0b6853ed3f1e767a36fdb4d7e6b24d72f246b114efb1f398"} Jul 15 19:38:55.494686 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:55.494258 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-ldd9p" event={"ID":"2edbfdc0-a0ec-4e17-897a-4e44954b19a9","Type":"ContainerStarted","Data":"6663ff1e3f556facb6ae192a4d429ca0e6efd0ffb2f502f2397882971374d96b"} Jul 15 19:38:55.496092 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:55.496068 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-87gx7" event={"ID":"bd5cd4f0-d31d-4f1f-877d-53537b0fc122","Type":"ContainerStarted","Data":"fa754b68f7b76024cd4b48f17297d062d4de094daacb50049b8615719b74b94d"} Jul 15 19:38:55.496183 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:55.496110 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-87gx7" event={"ID":"bd5cd4f0-d31d-4f1f-877d-53537b0fc122","Type":"ContainerStarted","Data":"7a617532e331b8649b77b3468bd2b9acf1c713ff51f1bcdf23d871c2ee790872"} Jul 15 19:38:55.514679 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:55.514615 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-ldd9p" podStartSLOduration=1.514594252 podStartE2EDuration="1.514594252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:55.513174813 +0000 UTC m=+49.757681275" watchObservedRunningTime="2026-07-15 19:38:55.514594252 +0000 UTC m=+49.759100715" Jul 15 19:38:55.540986 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:55.540921 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-87gx7" podStartSLOduration=1.797749276 podStartE2EDuration="3.540896791s" totalImagesPullingTime="1.743147515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:55.538110634 +0000 UTC m=+49.782617095" watchObservedRunningTime="2026-07-15 19:38:55.540896791 +0000 UTC m=+49.785403256" Jul 15 19:38:55.680103 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:55.680066 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:56.048832 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:56.048802 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wxt6l_2d86b57f-2938-41d5-8d6e-d2a17177030d/node-ca/0.log" Jul 15 19:38:56.448147 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:56.448115 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2288d_d896f9b8-775c-4130-820c-59831ff12641/serve-healthcheck-canary/0.log" Jul 15 19:38:56.595918 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:56.594492 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38c84af0-f6d3-4217-ba8a-87fb6912c97e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:56.595918 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:56.594634 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38c84af0-f6d3-4217-ba8a-87fb6912c97e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:38:56.595918 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:56.595834 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:38:56.595918 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:56.595894 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38c84af0-f6d3-4217-ba8a-87fb6912c97e-insights-runtime-extractor-tls podName:38c84af0-f6d3-4217-ba8a-87fb6912c97e nodeName:}" failed. No retries permitted until 2026-07-15 19:39:00.595875365 +0000 UTC m=+54.840381807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38c84af0-f6d3-4217-ba8a-87fb6912c97e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5rb4w" (UID: "38c84af0-f6d3-4217-ba8a-87fb6912c97e") : secret "insights-runtime-extractor-tls" not found Jul 15 19:38:56.596542 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:56.596152 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:38:56.596542 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:56.596214 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/38c84af0-f6d3-4217-ba8a-87fb6912c97e-kube-rbac-proxy-cm podName:38c84af0-f6d3-4217-ba8a-87fb6912c97e nodeName:}" failed. No retries permitted until 2026-07-15 19:39:00.596196759 +0000 UTC m=+54.840703207 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/38c84af0-f6d3-4217-ba8a-87fb6912c97e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5rb4w" (UID: "38c84af0-f6d3-4217-ba8a-87fb6912c97e") : configmap "kube-rbac-proxy" not found Jul 15 19:38:56.649716 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:56.649674 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-87gx7_bd5cd4f0-d31d-4f1f-877d-53537b0fc122/migrator/0.log" Jul 15 19:38:56.743539 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:56.743443 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:56.844977 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:56.844940 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:56.848947 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:56.848924 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-87gx7_bd5cd4f0-d31d-4f1f-877d-53537b0fc122/graceful-termination/0.log" Jul 15 19:38:57.051073 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:57.050981 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-69x22_2ae8e73f-008e-4977-8153-342f4a7460a5/kube-storage-version-migrator-operator/0.log" Jul 15 19:38:57.224672 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:57.224636 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:57.304253 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:57.304162 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:57.383131 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:57.383093 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:57.483669 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:57.483629 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:58.057013 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:58.056978 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:58.158577 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:58.158544 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:58.463289 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:58.463249 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:58.517470 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:58.517438 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b90b66a6-7fd1-47c8-8250-83f56348498c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-957d7\" (UID: \"b90b66a6-7fd1-47c8-8250-83f56348498c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7" Jul 15 19:38:58.517721 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:58.517536 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:38:58.517721 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:58.517573 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-fksq2\" (UID: \"88a9eb45-3875-4d7f-aab1-e1aeb90333d6\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" Jul 15 19:38:58.517721 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:58.517591 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:58.517721 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:58.517621 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:38:58.517721 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:58.517702 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 19:38:58.517721 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:58.517717 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d66974864-7mgdf: secret "image-registry-tls" not found Jul 15 19:38:58.518023 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:58.517726 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 15 19:38:58.518023 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:58.517781 2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 19:38:58.518023 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:58.517811 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls podName:72b9d425-51d4-4da9-8d59-bd071ec852bb nodeName:}" failed. No retries permitted until 2026-07-15 19:39:14.517792885 +0000 UTC m=+68.762299325 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls") pod "image-registry-6d66974864-7mgdf" (UID: "72b9d425-51d4-4da9-8d59-bd071ec852bb") : secret "image-registry-tls" not found Jul 15 19:38:58.518023 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:58.517839 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls podName:88a9eb45-3875-4d7f-aab1-e1aeb90333d6 nodeName:}" failed. No retries permitted until 2026-07-15 19:39:14.517818202 +0000 UTC m=+68.762324648 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-fksq2" (UID: "88a9eb45-3875-4d7f-aab1-e1aeb90333d6") : secret "cluster-monitoring-operator-tls" not found Jul 15 19:38:58.518023 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:58.517861 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs podName:eb9ebe84-37a6-4f73-880e-dad90c6453c7 nodeName:}" failed. No retries permitted until 2026-07-15 19:39:14.517851195 +0000 UTC m=+68.762357644 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs") pod "router-default-656f8d6d5b-5lzhw" (UID: "eb9ebe84-37a6-4f73-880e-dad90c6453c7") : secret "router-metrics-certs-default" not found Jul 15 19:38:58.518023 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:58.517877 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle podName:eb9ebe84-37a6-4f73-880e-dad90c6453c7 nodeName:}" failed. No retries permitted until 2026-07-15 19:39:14.517868114 +0000 UTC m=+68.762374569 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle") pod "router-default-656f8d6d5b-5lzhw" (UID: "eb9ebe84-37a6-4f73-880e-dad90c6453c7") : configmap references non-existent config key: service-ca.crt Jul 15 19:38:58.520036 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:58.520019 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b90b66a6-7fd1-47c8-8250-83f56348498c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-957d7\" (UID: \"b90b66a6-7fd1-47c8-8250-83f56348498c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7" Jul 15 19:38:58.570272 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:58.570242 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7" Jul 15 19:38:58.618727 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:58.618696 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls\") pod \"dns-default-zgrjv\" (UID: \"86437276-af54-49be-acf6-91f82d55e92c\") " pod="openshift-dns/dns-default-zgrjv" Jul 15 19:38:58.618883 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:58.618784 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-k8brr\" (UID: \"7a0eeb16-5201-43e8-a850-a7a3cdedc3a1\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k8brr" Jul 15 19:38:58.618883 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:58.618860 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:38:58.619004 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:58.618917 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 19:38:58.619004 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:58.618925 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls podName:86437276-af54-49be-acf6-91f82d55e92c nodeName:}" failed. No retries permitted until 2026-07-15 19:39:14.618909555 +0000 UTC m=+68.863415993 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls") pod "dns-default-zgrjv" (UID: "86437276-af54-49be-acf6-91f82d55e92c") : secret "dns-default-metrics-tls" not found Jul 15 19:38:58.619004 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:38:58.618970 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert podName:7a0eeb16-5201-43e8-a850-a7a3cdedc3a1 nodeName:}" failed. No retries permitted until 2026-07-15 19:39:14.618954439 +0000 UTC m=+68.863460894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-k8brr" (UID: "7a0eeb16-5201-43e8-a850-a7a3cdedc3a1") : secret "networking-console-plugin-cert" not found Jul 15 19:38:58.726407 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:58.726383 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7"] Jul 15 19:38:58.841988 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:58.841957 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:59.057341 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:59.057246 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:59.513104 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:59.513064 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7" event={"ID":"b90b66a6-7fd1-47c8-8250-83f56348498c","Type":"ContainerStarted","Data":"42fc80ab42a22ed93092eba37c61b4c681e39b9e2a786c8927a180f4e2b95d9d"} Jul 15 19:38:59.703736 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:59.703656 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:59.765319 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:38:59.765248 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:00.058709 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:00.058624 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:00.246625 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:00.246592 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:00.443684 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:00.443615 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:00.638796 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:00.638737 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38c84af0-f6d3-4217-ba8a-87fb6912c97e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:39:00.638955 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:00.638874 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38c84af0-f6d3-4217-ba8a-87fb6912c97e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:39:00.639026 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:39:00.638974 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:39:00.639073 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:39:00.639042 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38c84af0-f6d3-4217-ba8a-87fb6912c97e-insights-runtime-extractor-tls podName:38c84af0-f6d3-4217-ba8a-87fb6912c97e nodeName:}" failed. No retries permitted until 2026-07-15 19:39:08.639023154 +0000 UTC m=+62.883529594 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38c84af0-f6d3-4217-ba8a-87fb6912c97e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5rb4w" (UID: "38c84af0-f6d3-4217-ba8a-87fb6912c97e") : secret "insights-runtime-extractor-tls" not found Jul 15 19:39:00.639385 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:00.639363 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38c84af0-f6d3-4217-ba8a-87fb6912c97e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:39:00.644883 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:00.644854 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:00.960228 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:00.960190 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:01.521473 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:01.521434 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7" event={"ID":"b90b66a6-7fd1-47c8-8250-83f56348498c","Type":"ContainerStarted","Data":"eb981a3037f5dfa87057ebd11cd812de33a3df2148cfa721ed11877231bd4c08"} Jul 15 19:39:01.521873 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:01.521489 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7" event={"ID":"b90b66a6-7fd1-47c8-8250-83f56348498c","Type":"ContainerStarted","Data":"4ffa7e92ead8866f2ae96f14796da40ef978fb20611c89dc15acb4c316e910af"} Jul 15 19:39:01.544079 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:01.544027 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-957d7" podStartSLOduration=35.562678483 podStartE2EDuration="37.544009538s" totalImagesPullingTime="1.981331055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:39:01.543560095 +0000 UTC m=+55.788066556" watchObservedRunningTime="2026-07-15 19:39:01.544009538 +0000 UTC m=+55.788516023" Jul 15 19:39:01.645122 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:01.645087 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:01.748254 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:01.748214 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:01.851037 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:01.850937 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:01.952011 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:01.951977 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:02.247199 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:02.247154 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:02.441147 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:02.441099 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:02.643924 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:02.643835 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:03.691117 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:03.691078 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:03.765377 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:03.765342 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:04.208738 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:04.208700 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:04.309139 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:04.309095 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:04.450701 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:04.450664 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:04.846108 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:04.846070 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:05.444215 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:05.444178 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:05.845855 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:05.845730 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:06.241559 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:06.241519 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:06.404417 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:06.404382 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nxd2f" Jul 15 19:39:06.441522 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:06.441450 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:06.938027 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:06.937991 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:06.998234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:06.998203 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:07.069063 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:07.068995 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:07.233130 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:07.233060 2554 scope.go:117] "RemoveContainer" containerID="ce6d05724b8d9ad112e9ce5a9d7db0f3771fb5536965b5bb11a1308554214fcd" Jul 15 19:39:07.242166 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:07.242138 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:07.452594 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:07.452556 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:07.545534 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:07.545457 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 19:39:07.545534 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:07.545525 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-2ldzn" event={"ID":"b21f8771-ffd5-4487-812b-bfbaa7718461","Type":"ContainerStarted","Data":"94090afa6f95a8f60af334f836565b3dc154a809a7a5b09cc52bf7deffb9ba7f"} Jul 15 19:39:07.545830 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:07.545805 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-2ldzn" Jul 15 19:39:07.565660 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:07.565613 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-2ldzn" podStartSLOduration=36.287121528 podStartE2EDuration="43.565598852s" totalImagesPullingTime="7.278477324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:39:07.564229274 +0000 UTC m=+61.808735735" watchObservedRunningTime="2026-07-15 19:39:07.565598852 +0000 UTC m=+61.810105312" Jul 15 19:39:07.779365 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:07.779336 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-2ldzn" Jul 15 19:39:07.959409 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:07.959361 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:08.060000 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:08.059958 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:08.163516 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:08.163482 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:08.269428 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:08.269347 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:08.643154 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:08.643069 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:08.710286 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:08.710251 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38c84af0-f6d3-4217-ba8a-87fb6912c97e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:39:08.712613 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:08.712585 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38c84af0-f6d3-4217-ba8a-87fb6912c97e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rb4w\" (UID: \"38c84af0-f6d3-4217-ba8a-87fb6912c97e\") " pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:39:08.718796 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:08.718744 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:08.784543 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:08.784516 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hjcsl\"" Jul 15 19:39:08.792275 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:08.792247 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5rb4w" Jul 15 19:39:08.922634 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:08.922579 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5rb4w"] Jul 15 19:39:08.924658 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:39:08.924631 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c84af0_f6d3_4217_ba8a_87fb6912c97e.slice/crio-1aae163d70c3f879dc1dc27a491260c11a1cd2dcd24a36496661f7406229382f WatchSource:0}: Error finding container 1aae163d70c3f879dc1dc27a491260c11a1cd2dcd24a36496661f7406229382f: Status 404 returned error can't find the container with id 1aae163d70c3f879dc1dc27a491260c11a1cd2dcd24a36496661f7406229382f Jul 15 19:39:09.203326 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:09.203248 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:09.304170 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:09.304129 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:09.449855 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:09.449824 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:09.522078 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:09.521792 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:09.554940 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:09.554827 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5rb4w" event={"ID":"38c84af0-f6d3-4217-ba8a-87fb6912c97e","Type":"ContainerStarted","Data":"1aae163d70c3f879dc1dc27a491260c11a1cd2dcd24a36496661f7406229382f"} Jul 15 19:39:09.656498 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:09.656420 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:09.897815 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:09.897710 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:09.978695 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:09.978611 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:10.054196 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:10.054163 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:10.297132 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:10.297089 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:11.270178 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:11.270141 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:11.943057 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:11.943023 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret\") pod \"global-pull-secret-syncer-rv9rh\" (UID: \"87560cf2-a9d6-46e4-86dd-eac6f20cc8c8\") " pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:39:11.943251 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:11.943197 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs\") pod \"network-metrics-daemon-477vh\" (UID: \"33e0c74e-e752-4c0f-9c26-a9742b866f41\") " pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:39:11.945953 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:11.945933 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 19:39:11.946120 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:11.945937 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 19:39:11.956769 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:11.956717 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87560cf2-a9d6-46e4-86dd-eac6f20cc8c8-original-pull-secret\") pod \"global-pull-secret-syncer-rv9rh\" (UID: \"87560cf2-a9d6-46e4-86dd-eac6f20cc8c8\") " pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:39:11.956898 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:11.956832 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e0c74e-e752-4c0f-9c26-a9742b866f41-metrics-certs\") pod \"network-metrics-daemon-477vh\" (UID: \"33e0c74e-e752-4c0f-9c26-a9742b866f41\") " pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:39:12.043684 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:12.043647 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzvq\" (UniqueName: \"kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq\") pod \"network-check-target-2ws6h\" (UID: \"9db8df3b-ca71-41fb-9039-a9b1afc9f1bd\") " pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:39:12.046672 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:12.046638 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfzvq\" (UniqueName: \"kubernetes.io/projected/9db8df3b-ca71-41fb-9039-a9b1afc9f1bd-kube-api-access-jfzvq\") pod \"network-check-target-2ws6h\" (UID: \"9db8df3b-ca71-41fb-9039-a9b1afc9f1bd\") " pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:39:12.058111 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:12.058086 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lw97b\"" Jul 15 19:39:12.065630 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:12.065611 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:39:12.066199 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:12.066180 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vqmsl\"" Jul 15 19:39:12.072685 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:12.072666 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv9rh" Jul 15 19:39:12.073734 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:12.073710 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-477vh" Jul 15 19:39:14.260016 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.259984 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rv9rh"] Jul 15 19:39:14.262542 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:39:14.262510 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87560cf2_a9d6_46e4_86dd_eac6f20cc8c8.slice/crio-3d69be5673e11845c7ca5c02391a0a53244c7f334a70321553014282aafc9c3f WatchSource:0}: Error finding container 3d69be5673e11845c7ca5c02391a0a53244c7f334a70321553014282aafc9c3f: Status 404 returned error can't find the container with id 3d69be5673e11845c7ca5c02391a0a53244c7f334a70321553014282aafc9c3f Jul 15 19:39:14.278078 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.278051 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-477vh"] Jul 15 19:39:14.281563 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:39:14.281535 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e0c74e_e752_4c0f_9c26_a9742b866f41.slice/crio-591f5604e880f794264f3fc7cea0330b52ffe6e8f941b64b21965102ff1efc39 WatchSource:0}: Error finding container 591f5604e880f794264f3fc7cea0330b52ffe6e8f941b64b21965102ff1efc39: Status 404 returned error can't find the container with id 591f5604e880f794264f3fc7cea0330b52ffe6e8f941b64b21965102ff1efc39 Jul 15 19:39:14.483261 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.483228 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2ws6h"] Jul 15 19:39:14.486977 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:39:14.486947 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db8df3b_ca71_41fb_9039_a9b1afc9f1bd.slice/crio-29b07b575481dda0bdd255d7eae57a245d4fb065afc68b6c3c87f5bd4af9ebf1 WatchSource:0}: Error finding container 29b07b575481dda0bdd255d7eae57a245d4fb065afc68b6c3c87f5bd4af9ebf1: Status 404 returned error can't find the container with id 29b07b575481dda0bdd255d7eae57a245d4fb065afc68b6c3c87f5bd4af9ebf1 Jul 15 19:39:14.566661 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.566633 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:39:14.566806 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.566690 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-fksq2\" (UID: \"88a9eb45-3875-4d7f-aab1-e1aeb90333d6\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" Jul 15 19:39:14.566806 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.566719 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:39:14.566806 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.566770 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:39:14.567629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.567512 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb9ebe84-37a6-4f73-880e-dad90c6453c7-service-ca-bundle\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:39:14.569456 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.569437 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/88a9eb45-3875-4d7f-aab1-e1aeb90333d6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-fksq2\" (UID: \"88a9eb45-3875-4d7f-aab1-e1aeb90333d6\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" Jul 15 19:39:14.569586 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.569564 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls\") pod \"image-registry-6d66974864-7mgdf\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:39:14.569662 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.569621 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb9ebe84-37a6-4f73-880e-dad90c6453c7-metrics-certs\") pod \"router-default-656f8d6d5b-5lzhw\" (UID: \"eb9ebe84-37a6-4f73-880e-dad90c6453c7\") " pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:39:14.578898 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.578868 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-477vh" event={"ID":"33e0c74e-e752-4c0f-9c26-a9742b866f41","Type":"ContainerStarted","Data":"591f5604e880f794264f3fc7cea0330b52ffe6e8f941b64b21965102ff1efc39"} Jul 15 19:39:14.580062 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.580036 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rv9rh" event={"ID":"87560cf2-a9d6-46e4-86dd-eac6f20cc8c8","Type":"ContainerStarted","Data":"3d69be5673e11845c7ca5c02391a0a53244c7f334a70321553014282aafc9c3f"} Jul 15 19:39:14.581476 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.581442 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2ws6h" event={"ID":"9db8df3b-ca71-41fb-9039-a9b1afc9f1bd","Type":"ContainerStarted","Data":"29b07b575481dda0bdd255d7eae57a245d4fb065afc68b6c3c87f5bd4af9ebf1"} Jul 15 19:39:14.667243 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.667206 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls\") pod \"dns-default-zgrjv\" (UID: \"86437276-af54-49be-acf6-91f82d55e92c\") " pod="openshift-dns/dns-default-zgrjv" Jul 15 19:39:14.667399 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.667262 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-k8brr\" (UID: \"7a0eeb16-5201-43e8-a850-a7a3cdedc3a1\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k8brr" Jul 15 19:39:14.669745 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.669718 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86437276-af54-49be-acf6-91f82d55e92c-metrics-tls\") pod \"dns-default-zgrjv\" (UID: \"86437276-af54-49be-acf6-91f82d55e92c\") " pod="openshift-dns/dns-default-zgrjv" Jul 15 19:39:14.669917 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.669898 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a0eeb16-5201-43e8-a850-a7a3cdedc3a1-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-k8brr\" (UID: \"7a0eeb16-5201-43e8-a850-a7a3cdedc3a1\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k8brr" Jul 15 19:39:14.701269 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.701236 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:14.798035 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.797948 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2tjcx\"" Jul 15 19:39:14.806123 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.806101 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:39:14.821093 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.821065 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vdnlr\"" Jul 15 19:39:14.829357 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.829338 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" Jul 15 19:39:14.842459 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.842435 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ngg2l\"" Jul 15 19:39:14.848532 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.848503 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:39:14.866418 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.866303 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-mlvd8\"" Jul 15 19:39:14.874458 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.874431 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-k8brr" Jul 15 19:39:14.880843 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.880785 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2fvq2\"" Jul 15 19:39:14.888876 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:14.888851 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zgrjv" Jul 15 19:39:15.002493 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.002415 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d66974864-7mgdf"] Jul 15 19:39:15.010005 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:39:15.009959 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b9d425_51d4_4da9_8d59_bd071ec852bb.slice/crio-84486036be31a5d1696d1c8fb9c413916fc9f55796e8d4dc86010f9eae7cabdf WatchSource:0}: Error finding container 84486036be31a5d1696d1c8fb9c413916fc9f55796e8d4dc86010f9eae7cabdf: Status 404 returned error can't find the container with id 84486036be31a5d1696d1c8fb9c413916fc9f55796e8d4dc86010f9eae7cabdf Jul 15 19:39:15.016697 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.016664 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2"] Jul 15 19:39:15.025180 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:39:15.025145 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a9eb45_3875_4d7f_aab1_e1aeb90333d6.slice/crio-8da3c88814893048940abac45a3996cfdc13b803c4357de9f639d785fdefd81c WatchSource:0}: Error finding container 8da3c88814893048940abac45a3996cfdc13b803c4357de9f639d785fdefd81c: Status 404 returned error can't find the container with id 8da3c88814893048940abac45a3996cfdc13b803c4357de9f639d785fdefd81c Jul 15 19:39:15.069226 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.069193 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-656f8d6d5b-5lzhw"] Jul 15 19:39:15.073410 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:39:15.073375 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9ebe84_37a6_4f73_880e_dad90c6453c7.slice/crio-2dc911f216d52f92365b9a228423665ef00e7048c8617a018d695522db352ed3 WatchSource:0}: Error finding container 2dc911f216d52f92365b9a228423665ef00e7048c8617a018d695522db352ed3: Status 404 returned error can't find the container with id 2dc911f216d52f92365b9a228423665ef00e7048c8617a018d695522db352ed3 Jul 15 19:39:15.101421 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.101320 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-k8brr"] Jul 15 19:39:15.107402 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:39:15.107371 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0eeb16_5201_43e8_a850_a7a3cdedc3a1.slice/crio-ca5a221b3eced45092b54a1b1ec3f7bce31ed8ed63bd428893ca62ea50b8b338 WatchSource:0}: Error finding container ca5a221b3eced45092b54a1b1ec3f7bce31ed8ed63bd428893ca62ea50b8b338: Status 404 returned error can't find the container with id ca5a221b3eced45092b54a1b1ec3f7bce31ed8ed63bd428893ca62ea50b8b338 Jul 15 19:39:15.136787 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.136738 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zgrjv"] Jul 15 19:39:15.139500 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:39:15.139457 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86437276_af54_49be_acf6_91f82d55e92c.slice/crio-419a5e0d918c45b52a6c5092c84bcd501c7f43be81e9c45a985db927459e4e18 WatchSource:0}: Error finding container 419a5e0d918c45b52a6c5092c84bcd501c7f43be81e9c45a985db927459e4e18: Status 404 returned error can't find the container with id 419a5e0d918c45b52a6c5092c84bcd501c7f43be81e9c45a985db927459e4e18 Jul 15 19:39:15.587700 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.587660 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" event={"ID":"88a9eb45-3875-4d7f-aab1-e1aeb90333d6","Type":"ContainerStarted","Data":"8da3c88814893048940abac45a3996cfdc13b803c4357de9f639d785fdefd81c"} Jul 15 19:39:15.590805 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.590307 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d66974864-7mgdf" event={"ID":"72b9d425-51d4-4da9-8d59-bd071ec852bb","Type":"ContainerStarted","Data":"b3613085d9e337b0b6b4699781a19138b5caeb95725b66d166b5186a19d25fc9"} Jul 15 19:39:15.590805 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.590364 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d66974864-7mgdf" event={"ID":"72b9d425-51d4-4da9-8d59-bd071ec852bb","Type":"ContainerStarted","Data":"84486036be31a5d1696d1c8fb9c413916fc9f55796e8d4dc86010f9eae7cabdf"} Jul 15 19:39:15.590805 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.590422 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:39:15.592564 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.592535 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5rb4w" event={"ID":"38c84af0-f6d3-4217-ba8a-87fb6912c97e","Type":"ContainerStarted","Data":"11d2b4f006fac15dfe41f84c50d37e26b75e33fbce2623fe801cbb7cdc90c264"} Jul 15 19:39:15.595809 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.595780 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" event={"ID":"eb9ebe84-37a6-4f73-880e-dad90c6453c7","Type":"ContainerStarted","Data":"14387a76b99c5dbc0b27309963ae8c316a4748d1348324a177411ef99832f645"} Jul 15 19:39:15.595920 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.595819 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" event={"ID":"eb9ebe84-37a6-4f73-880e-dad90c6453c7","Type":"ContainerStarted","Data":"2dc911f216d52f92365b9a228423665ef00e7048c8617a018d695522db352ed3"} Jul 15 19:39:15.597584 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.597553 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-k8brr" event={"ID":"7a0eeb16-5201-43e8-a850-a7a3cdedc3a1","Type":"ContainerStarted","Data":"ca5a221b3eced45092b54a1b1ec3f7bce31ed8ed63bd428893ca62ea50b8b338"} Jul 15 19:39:15.599421 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.599395 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2ws6h" event={"ID":"9db8df3b-ca71-41fb-9039-a9b1afc9f1bd","Type":"ContainerStarted","Data":"1fab4ee9186a7994e2ac075a47da3d2484a32678bc504ee7dce3059b6880e6c3"} Jul 15 19:39:15.599587 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.599549 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:39:15.600861 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.600839 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zgrjv" event={"ID":"86437276-af54-49be-acf6-91f82d55e92c","Type":"ContainerStarted","Data":"419a5e0d918c45b52a6c5092c84bcd501c7f43be81e9c45a985db927459e4e18"} Jul 15 19:39:15.611550 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.611058 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d66974864-7mgdf" podStartSLOduration=68.611039068 podStartE2EDuration="1m8.611039068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:39:15.610572878 +0000 UTC m=+69.855079341" watchObservedRunningTime="2026-07-15 19:39:15.611039068 +0000 UTC m=+69.855545525" Jul 15 19:39:15.632738 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.632680 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" podStartSLOduration=50.632662205 podStartE2EDuration="50.632662205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:39:15.631742005 +0000 UTC m=+69.876248466" watchObservedRunningTime="2026-07-15 19:39:15.632662205 +0000 UTC m=+69.877168666" Jul 15 19:39:15.649294 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.649237 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2ws6h" podStartSLOduration=69.649226751 podStartE2EDuration="1m9.649226751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:39:15.64846168 +0000 UTC m=+69.892968143" watchObservedRunningTime="2026-07-15 19:39:15.649226751 +0000 UTC m=+69.893733229" Jul 15 19:39:15.848984 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.848889 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:39:15.852857 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:15.852829 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:39:16.616123 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:16.616088 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-477vh" event={"ID":"33e0c74e-e752-4c0f-9c26-a9742b866f41","Type":"ContainerStarted","Data":"681423f4cd963fea01a71d69a9978e7f9d212d3f35154125076337ddcbb85076"} Jul 15 19:39:16.616582 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:16.616146 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-477vh" event={"ID":"33e0c74e-e752-4c0f-9c26-a9742b866f41","Type":"ContainerStarted","Data":"36bc05aef168dfd2476f54f45a90aa0aad3ae57bf01b73136d8cd45df1b55aea"} Jul 15 19:39:16.617147 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:16.617123 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:39:16.619883 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:16.619855 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-656f8d6d5b-5lzhw" Jul 15 19:39:16.632694 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:16.632640 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-477vh" podStartSLOduration=68.933685887 podStartE2EDuration="1m10.632620809s" totalImagesPullingTime="1.698934922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:39:16.632193106 +0000 UTC m=+70.876699568" watchObservedRunningTime="2026-07-15 19:39:16.632620809 +0000 UTC m=+70.877127273" Jul 15 19:39:17.315974 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:17.315912 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:17.456667 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:17.456589 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:17.552988 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:17.552887 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:17.648789 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:17.648685 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:17.754031 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:17.753967 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:17.846010 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:17.845922 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:18.626639 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:18.626558 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-k8brr" event={"ID":"7a0eeb16-5201-43e8-a850-a7a3cdedc3a1","Type":"ContainerStarted","Data":"365ba7f71df76e64b753e68480c7ed304149e1074954a5a69577fe4f7af2a7b7"} Jul 15 19:39:18.644554 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:18.644501 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-k8brr" podStartSLOduration=35.010349042 podStartE2EDuration="37.644485919s" totalImagesPullingTime="2.634136877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:39:18.643132396 +0000 UTC m=+72.887638857" watchObservedRunningTime="2026-07-15 19:39:18.644485919 +0000 UTC m=+72.888992379" Jul 15 19:39:18.776512 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:18.776471 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:18.848265 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:18.848234 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:21.659121 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:21.659044 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:21.760998 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:21.760955 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:21.933396 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:21.933353 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:21.993399 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:21.993358 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:22.723568 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:22.723530 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:22.802057 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:22.801994 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:23.647433 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:23.647399 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zgrjv" event={"ID":"86437276-af54-49be-acf6-91f82d55e92c","Type":"ContainerStarted","Data":"e188977027f94859c77bf6176874d692ac4ee5354ea4c6aa72217f1c1f660bff"} Jul 15 19:39:23.647608 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:23.647507 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zgrjv" event={"ID":"86437276-af54-49be-acf6-91f82d55e92c","Type":"ContainerStarted","Data":"65f96f4920ed052d939ddad2783fe5d2a4525b309a0f3c5399bf55f412e8b016"} Jul 15 19:39:23.647608 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:23.647579 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zgrjv" Jul 15 19:39:23.649059 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:23.649024 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rv9rh" event={"ID":"87560cf2-a9d6-46e4-86dd-eac6f20cc8c8","Type":"ContainerStarted","Data":"80f07ea194e0a830546f5352a1f86eba6af52b8f1ad905db6676ceaa149f3645"} Jul 15 19:39:23.650736 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:23.650708 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" event={"ID":"88a9eb45-3875-4d7f-aab1-e1aeb90333d6","Type":"ContainerStarted","Data":"048a5e876e75e6673b779355aebea5756106df201b6461cf567696dbc1b7ac2b"} Jul 15 19:39:23.652700 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:23.652678 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5rb4w" event={"ID":"38c84af0-f6d3-4217-ba8a-87fb6912c97e","Type":"ContainerStarted","Data":"52a6bbe6e8151638c2fc5afffd8fe63f79f6ff7f1d744a18f0b0780176deca38"} Jul 15 19:39:23.676152 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:23.676092 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zgrjv" podStartSLOduration=33.456902527 podStartE2EDuration="41.676073031s" totalImagesPullingTime="8.219170504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:39:23.675405633 +0000 UTC m=+77.919912100" watchObservedRunningTime="2026-07-15 19:39:23.676073031 +0000 UTC m=+77.920579494" Jul 15 19:39:23.700821 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:23.700738 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rv9rh" podStartSLOduration=68.603922189 podStartE2EDuration="1m17.700720772s" totalImagesPullingTime="9.096798583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:39:23.700066681 +0000 UTC m=+77.944573142" watchObservedRunningTime="2026-07-15 19:39:23.700720772 +0000 UTC m=+77.945227233" Jul 15 19:39:23.726221 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:23.726168 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fksq2" podStartSLOduration=51.392163909 podStartE2EDuration="59.726150097s" totalImagesPullingTime="8.333986188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:39:23.724142885 +0000 UTC m=+77.968649345" watchObservedRunningTime="2026-07-15 19:39:23.726150097 +0000 UTC m=+77.970656563" Jul 15 19:39:23.792198 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:23.792158 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:23.889360 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:23.889273 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:24.144098 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:24.144001 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:24.242176 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:24.242138 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:24.308937 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:24.308896 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:26.273807 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:26.273772 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:26.374218 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:26.374163 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:27.195689 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:27.195536 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:27.295605 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:27.295563 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:27.793932 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:27.793894 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:27.894240 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:27.894209 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:28.675085 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:28.675048 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5rb4w" event={"ID":"38c84af0-f6d3-4217-ba8a-87fb6912c97e","Type":"ContainerStarted","Data":"0109f8806b47e0da8214654d32b2fcef2ebc51f8384b2f706d2a0fd180fc6162"} Jul 15 19:39:28.694728 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:28.694672 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5rb4w" podStartSLOduration=17.793661975 podStartE2EDuration="36.694655499s" totalImagesPullingTime="18.900993524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:38:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:39:28.694157248 +0000 UTC m=+82.938663711" watchObservedRunningTime="2026-07-15 19:39:28.694655499 +0000 UTC m=+82.939161962" Jul 15 19:39:29.301400 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:29.301363 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:29.401436 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:29.401396 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:29.507811 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:29.507771 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:29.602292 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:29.602212 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:31.041525 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:31.041485 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:31.133566 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:31.133531 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:31.373352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:31.373264 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:32.836388 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:32.836330 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:32.905407 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:32.905351 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:33.305228 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:33.305178 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:33.365052 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:33.365011 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:33.658775 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:33.658674 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zgrjv" Jul 15 19:39:36.360676 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:36.360635 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:36.459286 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:36.459250 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:36.620163 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:36.620097 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:39:36.795587 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:36.795549 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:36.896610 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:36.896510 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:36.976285 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:36.976251 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:37.043529 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:37.043499 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:37.144092 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:37.144055 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:38.563882 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:38.563830 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:38.664738 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:38.664701 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:38.727908 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:38.727871 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:39.198596 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.198557 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:39.310573 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.310540 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:39.437248 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.437213 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:39.447622 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.447591 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7nqwk"] Jul 15 19:39:39.453991 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.453928 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.457492 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.457466 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 19:39:39.457717 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.457490 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 19:39:39.457889 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.457497 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 19:39:39.457962 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.457506 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fjfgc\"" Jul 15 19:39:39.458040 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.457516 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 19:39:39.470318 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.470290 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/caa0af65-db73-412a-9770-c8d23a985201-metrics-client-ca\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.470511 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.470489 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/caa0af65-db73-412a-9770-c8d23a985201-node-exporter-tls\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.470636 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.470539 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvrmz\" (UniqueName: \"kubernetes.io/projected/caa0af65-db73-412a-9770-c8d23a985201-kube-api-access-cvrmz\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.470636 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.470571 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/caa0af65-db73-412a-9770-c8d23a985201-root\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.470777 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.470698 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/caa0af65-db73-412a-9770-c8d23a985201-node-exporter-textfile\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.470777 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.470744 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/caa0af65-db73-412a-9770-c8d23a985201-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.470890 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.470841 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/caa0af65-db73-412a-9770-c8d23a985201-node-exporter-wtmp\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.471399 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.471374 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caa0af65-db73-412a-9770-c8d23a985201-sys\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.471493 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.471434 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/caa0af65-db73-412a-9770-c8d23a985201-node-exporter-accelerators-collector-config\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.568207 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.568157 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:39.572187 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.572152 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/caa0af65-db73-412a-9770-c8d23a985201-node-exporter-textfile\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.572351 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.572196 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/caa0af65-db73-412a-9770-c8d23a985201-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.572351 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.572232 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/caa0af65-db73-412a-9770-c8d23a985201-node-exporter-wtmp\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.572351 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.572269 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caa0af65-db73-412a-9770-c8d23a985201-sys\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.572351 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.572301 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/caa0af65-db73-412a-9770-c8d23a985201-node-exporter-accelerators-collector-config\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.572351 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.572335 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/caa0af65-db73-412a-9770-c8d23a985201-metrics-client-ca\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.572620 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.572365 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/caa0af65-db73-412a-9770-c8d23a985201-node-exporter-tls\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.572620 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.572391 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvrmz\" (UniqueName: \"kubernetes.io/projected/caa0af65-db73-412a-9770-c8d23a985201-kube-api-access-cvrmz\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.572620 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.572411 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/caa0af65-db73-412a-9770-c8d23a985201-root\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.572620 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.572439 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/caa0af65-db73-412a-9770-c8d23a985201-node-exporter-wtmp\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.572620 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.572441 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caa0af65-db73-412a-9770-c8d23a985201-sys\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.572620 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.572491 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/caa0af65-db73-412a-9770-c8d23a985201-root\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.572620 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.572560 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/caa0af65-db73-412a-9770-c8d23a985201-node-exporter-textfile\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.572956 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:39:39.572646 2554 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 15 19:39:39.572956 ip-10-0-135-139 kubenswrapper[2554]: E0715 19:39:39.572715 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caa0af65-db73-412a-9770-c8d23a985201-node-exporter-tls podName:caa0af65-db73-412a-9770-c8d23a985201 nodeName:}" failed. No retries permitted until 2026-07-15 19:39:40.072695352 +0000 UTC m=+94.317201798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/caa0af65-db73-412a-9770-c8d23a985201-node-exporter-tls") pod "node-exporter-7nqwk" (UID: "caa0af65-db73-412a-9770-c8d23a985201") : secret "node-exporter-tls" not found Jul 15 19:39:39.573046 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.573006 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/caa0af65-db73-412a-9770-c8d23a985201-metrics-client-ca\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.573113 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.573089 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/caa0af65-db73-412a-9770-c8d23a985201-node-exporter-accelerators-collector-config\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.574872 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.574850 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/caa0af65-db73-412a-9770-c8d23a985201-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.592798 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.592746 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvrmz\" (UniqueName: \"kubernetes.io/projected/caa0af65-db73-412a-9770-c8d23a985201-kube-api-access-cvrmz\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:39.655542 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.655505 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:39.722192 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.722104 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:39.798158 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:39.798121 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:40.077220 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:40.077126 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/caa0af65-db73-412a-9770-c8d23a985201-node-exporter-tls\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:40.079692 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:40.079668 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/caa0af65-db73-412a-9770-c8d23a985201-node-exporter-tls\") pod \"node-exporter-7nqwk\" (UID: \"caa0af65-db73-412a-9770-c8d23a985201\") " pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:40.137104 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:40.137073 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:40.313085 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:40.313051 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:40.371596 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:40.371510 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7nqwk" Jul 15 19:39:40.386050 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:39:40.386009 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa0af65_db73_412a_9770_c8d23a985201.slice/crio-1cd3e64fca8f9d590521f58312eb1173dca955bbc7d05cdf8aaa94f4232aadf9 WatchSource:0}: Error finding container 1cd3e64fca8f9d590521f58312eb1173dca955bbc7d05cdf8aaa94f4232aadf9: Status 404 returned error can't find the container with id 1cd3e64fca8f9d590521f58312eb1173dca955bbc7d05cdf8aaa94f4232aadf9 Jul 15 19:39:40.423274 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:40.423243 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:40.508226 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:40.508187 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:40.632350 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:40.632257 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:40.724349 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:40.724318 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7nqwk" event={"ID":"caa0af65-db73-412a-9770-c8d23a985201","Type":"ContainerStarted","Data":"1cd3e64fca8f9d590521f58312eb1173dca955bbc7d05cdf8aaa94f4232aadf9"} Jul 15 19:39:41.663013 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:41.662933 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d66974864-7mgdf"] Jul 15 19:39:41.729326 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:41.729290 2554 generic.go:356] "Generic (PLEG): container finished" podID="caa0af65-db73-412a-9770-c8d23a985201" containerID="84e5e5244d4da2325545f91110912090bba9efa437cc37343207cc0d73abc757" exitCode=0 Jul 15 19:39:41.729497 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:41.729378 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7nqwk" event={"ID":"caa0af65-db73-412a-9770-c8d23a985201","Type":"ContainerDied","Data":"84e5e5244d4da2325545f91110912090bba9efa437cc37343207cc0d73abc757"} Jul 15 19:39:41.763005 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:41.762978 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:41.834101 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:41.834073 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:41.991402 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:41.991365 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:42.058445 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:42.058405 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:42.356847 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:42.356730 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:42.438299 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:42.438261 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:42.533406 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:42.533363 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:42.633199 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:42.633101 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:42.735563 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:42.735521 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7nqwk" event={"ID":"caa0af65-db73-412a-9770-c8d23a985201","Type":"ContainerStarted","Data":"6d8ce9a14cd0bc1f5ad483e9ee22d6ca3fd433b1f258e32a0955dc05f3b78cdc"} Jul 15 19:39:42.735952 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:42.735582 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7nqwk" event={"ID":"caa0af65-db73-412a-9770-c8d23a985201","Type":"ContainerStarted","Data":"6904e119f1b1b7967eeace062d29f509b4c406627737749dd26713d706d7ae64"} Jul 15 19:39:42.758620 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:42.758573 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7nqwk" podStartSLOduration=2.851560015 podStartE2EDuration="3.758556775s" totalImagesPullingTime="835.472056ms" totalInitContainerRuntime="71.524704ms" isStatefulPod=true podCreationTimestamp="2026-07-15 19:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:39:42.757834092 +0000 UTC m=+97.002340622" watchObservedRunningTime="2026-07-15 19:39:42.758556775 +0000 UTC m=+97.003063237" Jul 15 19:39:42.996621 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:42.996592 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:43.291667 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:43.291544 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:44.203026 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:44.202989 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:44.283911 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:44.283858 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:44.372685 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:44.372643 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:44.454834 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:44.454732 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:44.658740 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:44.658704 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:45.550676 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:45.550642 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:45.653994 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:45.653910 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:45.723766 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:45.723726 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:46.167602 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:46.167572 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:46.268910 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:46.268859 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:46.620335 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:46.620306 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2ws6h" Jul 15 19:39:51.791887 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:51.791849 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:51.932407 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:51.932369 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:51.989897 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:51.989868 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:55.685303 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:55.685249 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:55.785266 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:55.785227 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:55.887094 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:55.887056 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:57.792031 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:57.792000 2554 generic.go:356] "Generic (PLEG): container finished" podID="e2691146-bc04-4c7b-bb0c-b3377762da6e" containerID="790482c7093bcc9c0a065aee3aa1e0445c3ae2561feebf7b42d91e0bd12896dd" exitCode=0 Jul 15 19:39:57.792451 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:57.792088 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" event={"ID":"e2691146-bc04-4c7b-bb0c-b3377762da6e","Type":"ContainerDied","Data":"790482c7093bcc9c0a065aee3aa1e0445c3ae2561feebf7b42d91e0bd12896dd"} Jul 15 19:39:57.792451 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:57.792436 2554 scope.go:117] "RemoveContainer" containerID="790482c7093bcc9c0a065aee3aa1e0445c3ae2561feebf7b42d91e0bd12896dd" Jul 15 19:39:58.798477 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:39:58.798448 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-pmncb" event={"ID":"e2691146-bc04-4c7b-bb0c-b3377762da6e","Type":"ContainerStarted","Data":"3e5133fb09d2515332d69607ddf883317c41992f8e075dc3df901c83a185f68d"} Jul 15 19:40:06.683999 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.683937 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6d66974864-7mgdf" podUID="72b9d425-51d4-4da9-8d59-bd071ec852bb" containerName="registry" containerID="cri-o://b3613085d9e337b0b6b4699781a19138b5caeb95725b66d166b5186a19d25fc9" gracePeriod=30 Jul 15 19:40:06.831384 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.831206 2554 generic.go:356] "Generic (PLEG): container finished" podID="72b9d425-51d4-4da9-8d59-bd071ec852bb" containerID="b3613085d9e337b0b6b4699781a19138b5caeb95725b66d166b5186a19d25fc9" exitCode=0 Jul 15 19:40:06.831384 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.831319 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d66974864-7mgdf" event={"ID":"72b9d425-51d4-4da9-8d59-bd071ec852bb","Type":"ContainerDied","Data":"b3613085d9e337b0b6b4699781a19138b5caeb95725b66d166b5186a19d25fc9"} Jul 15 19:40:06.938653 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.938565 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:40:06.938829 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.938686 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:40:06.987877 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.987843 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls\") pod \"72b9d425-51d4-4da9-8d59-bd071ec852bb\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " Jul 15 19:40:06.987877 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.987882 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/72b9d425-51d4-4da9-8d59-bd071ec852bb-image-registry-private-configuration\") pod \"72b9d425-51d4-4da9-8d59-bd071ec852bb\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " Jul 15 19:40:06.988118 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.987947 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/72b9d425-51d4-4da9-8d59-bd071ec852bb-installation-pull-secrets\") pod \"72b9d425-51d4-4da9-8d59-bd071ec852bb\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " Jul 15 19:40:06.988118 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.988036 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-bound-sa-token\") pod \"72b9d425-51d4-4da9-8d59-bd071ec852bb\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " Jul 15 19:40:06.988118 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.988088 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-certificates\") pod \"72b9d425-51d4-4da9-8d59-bd071ec852bb\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " Jul 15 19:40:06.988118 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.988109 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/72b9d425-51d4-4da9-8d59-bd071ec852bb-ca-trust-extracted\") pod \"72b9d425-51d4-4da9-8d59-bd071ec852bb\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " Jul 15 19:40:06.988326 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.988140 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j7sk\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-kube-api-access-7j7sk\") pod \"72b9d425-51d4-4da9-8d59-bd071ec852bb\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " Jul 15 19:40:06.988326 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.988178 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72b9d425-51d4-4da9-8d59-bd071ec852bb-trusted-ca\") pod \"72b9d425-51d4-4da9-8d59-bd071ec852bb\" (UID: \"72b9d425-51d4-4da9-8d59-bd071ec852bb\") " Jul 15 19:40:06.988811 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.988728 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b9d425-51d4-4da9-8d59-bd071ec852bb-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "72b9d425-51d4-4da9-8d59-bd071ec852bb" (UID: "72b9d425-51d4-4da9-8d59-bd071ec852bb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:40:06.988923 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.988746 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "72b9d425-51d4-4da9-8d59-bd071ec852bb" (UID: "72b9d425-51d4-4da9-8d59-bd071ec852bb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:40:06.990600 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.990565 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b9d425-51d4-4da9-8d59-bd071ec852bb-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "72b9d425-51d4-4da9-8d59-bd071ec852bb" (UID: "72b9d425-51d4-4da9-8d59-bd071ec852bb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:40:06.990716 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.990576 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "72b9d425-51d4-4da9-8d59-bd071ec852bb" (UID: "72b9d425-51d4-4da9-8d59-bd071ec852bb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:40:06.991122 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.991092 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "72b9d425-51d4-4da9-8d59-bd071ec852bb" (UID: "72b9d425-51d4-4da9-8d59-bd071ec852bb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:40:06.991212 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.991101 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b9d425-51d4-4da9-8d59-bd071ec852bb-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "72b9d425-51d4-4da9-8d59-bd071ec852bb" (UID: "72b9d425-51d4-4da9-8d59-bd071ec852bb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:40:06.991212 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.991178 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-kube-api-access-7j7sk" (OuterVolumeSpecName: "kube-api-access-7j7sk") pod "72b9d425-51d4-4da9-8d59-bd071ec852bb" (UID: "72b9d425-51d4-4da9-8d59-bd071ec852bb"). InnerVolumeSpecName "kube-api-access-7j7sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:40:06.997104 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:06.997076 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b9d425-51d4-4da9-8d59-bd071ec852bb-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "72b9d425-51d4-4da9-8d59-bd071ec852bb" (UID: "72b9d425-51d4-4da9-8d59-bd071ec852bb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:40:07.040931 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:07.040896 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:40:07.089725 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:07.089689 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j7sk\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-kube-api-access-7j7sk\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:40:07.089725 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:07.089720 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72b9d425-51d4-4da9-8d59-bd071ec852bb-trusted-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:40:07.089942 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:07.089734 2554 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-tls\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:40:07.089942 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:07.089746 2554 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/72b9d425-51d4-4da9-8d59-bd071ec852bb-image-registry-private-configuration\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:40:07.089942 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:07.089780 2554 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/72b9d425-51d4-4da9-8d59-bd071ec852bb-installation-pull-secrets\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:40:07.089942 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:07.089792 2554 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72b9d425-51d4-4da9-8d59-bd071ec852bb-bound-sa-token\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:40:07.089942 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:07.089803 2554 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/72b9d425-51d4-4da9-8d59-bd071ec852bb-registry-certificates\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:40:07.089942 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:07.089814 2554 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/72b9d425-51d4-4da9-8d59-bd071ec852bb-ca-trust-extracted\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:40:07.835765 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:07.835736 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d66974864-7mgdf" Jul 15 19:40:07.836165 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:07.835787 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d66974864-7mgdf" event={"ID":"72b9d425-51d4-4da9-8d59-bd071ec852bb","Type":"ContainerDied","Data":"84486036be31a5d1696d1c8fb9c413916fc9f55796e8d4dc86010f9eae7cabdf"} Jul 15 19:40:07.836165 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:07.835851 2554 scope.go:117] "RemoveContainer" containerID="b3613085d9e337b0b6b4699781a19138b5caeb95725b66d166b5186a19d25fc9" Jul 15 19:40:07.856688 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:07.856665 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d66974864-7mgdf"] Jul 15 19:40:07.860614 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:07.860588 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d66974864-7mgdf"] Jul 15 19:40:08.238162 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:08.238131 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72b9d425-51d4-4da9-8d59-bd071ec852bb" path="/var/lib/kubelet/pods/72b9d425-51d4-4da9-8d59-bd071ec852bb/volumes" Jul 15 19:40:11.856665 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:11.856576 2554 generic.go:356] "Generic (PLEG): container finished" podID="2ae8e73f-008e-4977-8153-342f4a7460a5" containerID="054c270ef7c29f645e8d1dc75bec1ab247e8b833d5d5dd8f4c0cf9ea8ea682b0" exitCode=0 Jul 15 19:40:11.856665 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:11.856644 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-69x22" event={"ID":"2ae8e73f-008e-4977-8153-342f4a7460a5","Type":"ContainerDied","Data":"054c270ef7c29f645e8d1dc75bec1ab247e8b833d5d5dd8f4c0cf9ea8ea682b0"} Jul 15 19:40:11.857160 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:11.857014 2554 scope.go:117] "RemoveContainer" containerID="054c270ef7c29f645e8d1dc75bec1ab247e8b833d5d5dd8f4c0cf9ea8ea682b0" Jul 15 19:40:11.858406 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:11.858384 2554 generic.go:356] "Generic (PLEG): container finished" podID="e3227b7b-2925-413d-aaf3-aa5e5bb60a81" containerID="068c68a4c5d0c989441d12f3fa3d9e35071434763040d1b7853c3a0d83627115" exitCode=0 Jul 15 19:40:11.858478 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:11.858426 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftq5p" event={"ID":"e3227b7b-2925-413d-aaf3-aa5e5bb60a81","Type":"ContainerDied","Data":"068c68a4c5d0c989441d12f3fa3d9e35071434763040d1b7853c3a0d83627115"} Jul 15 19:40:11.858732 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:11.858711 2554 scope.go:117] "RemoveContainer" containerID="068c68a4c5d0c989441d12f3fa3d9e35071434763040d1b7853c3a0d83627115" Jul 15 19:40:12.864286 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:12.864251 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-69x22" event={"ID":"2ae8e73f-008e-4977-8153-342f4a7460a5","Type":"ContainerStarted","Data":"f825c2b58a6ad90afe5ea6de4c1992906a1289c462a0764ce11e7033f74e466a"} Jul 15 19:40:12.866084 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:12.866059 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftq5p" event={"ID":"e3227b7b-2925-413d-aaf3-aa5e5bb60a81","Type":"ContainerStarted","Data":"b551b9c112bc3137e50bd0fbcc321fa51a5c676b47452f2b02c186b402543071"} Jul 15 19:40:32.687442 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:32.687408 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:40:58.725149 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:58.725051 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:40:58.973569 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:40:58.973530 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:07.805645 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:07.805580 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:07.907968 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:07.907931 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:13.947541 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:13.947499 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:14.011367 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:14.011329 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:18.288222 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:18.288171 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:18.349594 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:18.349558 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:18.613839 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:18.613734 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:18.714730 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:18.714697 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:19.219201 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:19.219165 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:19.278428 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:19.278394 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:20.222770 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:20.222721 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:20.323915 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:20.323879 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:25.914870 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:25.914828 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:25.990200 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:25.990169 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:28.889957 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:28.889916 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:28.953178 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:28.953144 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:29.374352 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:29.374320 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:29.469941 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:29.469906 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:36.950196 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:36.950164 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:37.051237 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:37.051196 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:40.105706 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:40.105674 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:40.198471 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:40.198442 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:49.965851 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:41:49.965813 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:21.872324 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:42:21.872282 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:21.950836 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:42:21.950805 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:22.026019 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:42:22.025984 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:22.102620 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:42:22.102582 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:29.256609 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:42:29.256505 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:29.357381 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:42:29.357343 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:43:06.172951 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:06.172924 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 19:43:06.174290 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:06.174267 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 19:43:06.178116 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:06.178095 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 19:43:06.179174 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:06.179152 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 19:43:06.183315 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:06.183299 2554 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 19:43:26.418321 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:26.418266 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:43:39.493431 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.493392 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x"] Jul 15 19:43:39.494058 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.493826 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b9d425-51d4-4da9-8d59-bd071ec852bb" containerName="registry" Jul 15 19:43:39.494058 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.493842 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b9d425-51d4-4da9-8d59-bd071ec852bb" containerName="registry" Jul 15 19:43:39.494058 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.493910 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="72b9d425-51d4-4da9-8d59-bd071ec852bb" containerName="registry" Jul 15 19:43:39.496956 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.496935 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x" Jul 15 19:43:39.499773 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.499737 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 19:43:39.500606 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.500591 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xw4wp\"" Jul 15 19:43:39.500659 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.500608 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 19:43:39.509921 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.509892 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x"] Jul 15 19:43:39.620980 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.620943 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9829fc4e-8496-46e1-9861-05365842c043-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x\" (UID: \"9829fc4e-8496-46e1-9861-05365842c043\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x" Jul 15 19:43:39.621158 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.621034 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9829fc4e-8496-46e1-9861-05365842c043-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x\" (UID: \"9829fc4e-8496-46e1-9861-05365842c043\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x" Jul 15 19:43:39.621158 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.621065 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d78dq\" (UniqueName: \"kubernetes.io/projected/9829fc4e-8496-46e1-9861-05365842c043-kube-api-access-d78dq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x\" (UID: \"9829fc4e-8496-46e1-9861-05365842c043\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x" Jul 15 19:43:39.722281 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.722242 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9829fc4e-8496-46e1-9861-05365842c043-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x\" (UID: \"9829fc4e-8496-46e1-9861-05365842c043\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x" Jul 15 19:43:39.722475 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.722292 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9829fc4e-8496-46e1-9861-05365842c043-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x\" (UID: \"9829fc4e-8496-46e1-9861-05365842c043\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x" Jul 15 19:43:39.722475 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.722324 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d78dq\" (UniqueName: \"kubernetes.io/projected/9829fc4e-8496-46e1-9861-05365842c043-kube-api-access-d78dq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x\" (UID: \"9829fc4e-8496-46e1-9861-05365842c043\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x" Jul 15 19:43:39.722696 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.722672 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9829fc4e-8496-46e1-9861-05365842c043-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x\" (UID: \"9829fc4e-8496-46e1-9861-05365842c043\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x" Jul 15 19:43:39.722792 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.722740 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9829fc4e-8496-46e1-9861-05365842c043-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x\" (UID: \"9829fc4e-8496-46e1-9861-05365842c043\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x" Jul 15 19:43:39.730894 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.730870 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d78dq\" (UniqueName: \"kubernetes.io/projected/9829fc4e-8496-46e1-9861-05365842c043-kube-api-access-d78dq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x\" (UID: \"9829fc4e-8496-46e1-9861-05365842c043\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x" Jul 15 19:43:39.806129 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.806042 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x" Jul 15 19:43:39.943152 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.942960 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x"] Jul 15 19:43:39.946255 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:43:39.946221 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9829fc4e_8496_46e1_9861_05365842c043.slice/crio-6924862148ff950d3708d62857fd561064bedb0440cdcadcd501ce44106005a3 WatchSource:0}: Error finding container 6924862148ff950d3708d62857fd561064bedb0440cdcadcd501ce44106005a3: Status 404 returned error can't find the container with id 6924862148ff950d3708d62857fd561064bedb0440cdcadcd501ce44106005a3 Jul 15 19:43:39.948429 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:39.948413 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:43:40.629937 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:40.629890 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x" event={"ID":"9829fc4e-8496-46e1-9861-05365842c043","Type":"ContainerStarted","Data":"6924862148ff950d3708d62857fd561064bedb0440cdcadcd501ce44106005a3"} Jul 15 19:43:47.659718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:47.659679 2554 generic.go:356] "Generic (PLEG): container finished" podID="9829fc4e-8496-46e1-9861-05365842c043" containerID="580493bdb39962f27ae4b77538b14bfa87f3edddfe4df6b0148d7242ed08d55c" exitCode=0 Jul 15 19:43:47.660116 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:47.659779 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x" event={"ID":"9829fc4e-8496-46e1-9861-05365842c043","Type":"ContainerDied","Data":"580493bdb39962f27ae4b77538b14bfa87f3edddfe4df6b0148d7242ed08d55c"} Jul 15 19:43:49.669541 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:49.669505 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x" event={"ID":"9829fc4e-8496-46e1-9861-05365842c043","Type":"ContainerStarted","Data":"899331b1107e5c4523607a7a4763d93565d2b2bbb152ad3e3cf18e453f9b3e3d"} Jul 15 19:43:50.675168 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:50.675128 2554 generic.go:356] "Generic (PLEG): container finished" podID="9829fc4e-8496-46e1-9861-05365842c043" containerID="899331b1107e5c4523607a7a4763d93565d2b2bbb152ad3e3cf18e453f9b3e3d" exitCode=0 Jul 15 19:43:50.675535 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:50.675209 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x" event={"ID":"9829fc4e-8496-46e1-9861-05365842c043","Type":"ContainerDied","Data":"899331b1107e5c4523607a7a4763d93565d2b2bbb152ad3e3cf18e453f9b3e3d"} Jul 15 19:43:59.712030 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:59.711997 2554 generic.go:356] "Generic (PLEG): container finished" podID="9829fc4e-8496-46e1-9861-05365842c043" containerID="c789a8adc1267c4e6a91f8d8654c1fe5800b2c5adfec060de63dfd6948f182f7" exitCode=0 Jul 15 19:43:59.712410 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:43:59.712037 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x" event={"ID":"9829fc4e-8496-46e1-9861-05365842c043","Type":"ContainerDied","Data":"c789a8adc1267c4e6a91f8d8654c1fe5800b2c5adfec060de63dfd6948f182f7"} Jul 15 19:44:00.842952 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:00.842924 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x" Jul 15 19:44:01.005234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:01.005143 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d78dq\" (UniqueName: \"kubernetes.io/projected/9829fc4e-8496-46e1-9861-05365842c043-kube-api-access-d78dq\") pod \"9829fc4e-8496-46e1-9861-05365842c043\" (UID: \"9829fc4e-8496-46e1-9861-05365842c043\") " Jul 15 19:44:01.005234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:01.005193 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9829fc4e-8496-46e1-9861-05365842c043-util\") pod \"9829fc4e-8496-46e1-9861-05365842c043\" (UID: \"9829fc4e-8496-46e1-9861-05365842c043\") " Jul 15 19:44:01.005433 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:01.005247 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9829fc4e-8496-46e1-9861-05365842c043-bundle\") pod \"9829fc4e-8496-46e1-9861-05365842c043\" (UID: \"9829fc4e-8496-46e1-9861-05365842c043\") " Jul 15 19:44:01.006019 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:01.005988 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9829fc4e-8496-46e1-9861-05365842c043-bundle" (OuterVolumeSpecName: "bundle") pod "9829fc4e-8496-46e1-9861-05365842c043" (UID: "9829fc4e-8496-46e1-9861-05365842c043"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:44:01.007524 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:01.007495 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9829fc4e-8496-46e1-9861-05365842c043-kube-api-access-d78dq" (OuterVolumeSpecName: "kube-api-access-d78dq") pod "9829fc4e-8496-46e1-9861-05365842c043" (UID: "9829fc4e-8496-46e1-9861-05365842c043"). InnerVolumeSpecName "kube-api-access-d78dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:44:01.009443 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:01.009423 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9829fc4e-8496-46e1-9861-05365842c043-util" (OuterVolumeSpecName: "util") pod "9829fc4e-8496-46e1-9861-05365842c043" (UID: "9829fc4e-8496-46e1-9861-05365842c043"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:44:01.106024 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:01.105986 2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9829fc4e-8496-46e1-9861-05365842c043-util\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:44:01.106024 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:01.106017 2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9829fc4e-8496-46e1-9861-05365842c043-bundle\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:44:01.106024 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:01.106031 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d78dq\" (UniqueName: \"kubernetes.io/projected/9829fc4e-8496-46e1-9861-05365842c043-kube-api-access-d78dq\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:44:01.720558 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:01.720525 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x" event={"ID":"9829fc4e-8496-46e1-9861-05365842c043","Type":"ContainerDied","Data":"6924862148ff950d3708d62857fd561064bedb0440cdcadcd501ce44106005a3"} Jul 15 19:44:01.720558 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:01.720560 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jvl7x" Jul 15 19:44:01.720776 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:01.720580 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6924862148ff950d3708d62857fd561064bedb0440cdcadcd501ce44106005a3" Jul 15 19:44:05.642249 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:05.642205 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:06.434059 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:06.434025 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:07.571671 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.571631 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jjtqm"] Jul 15 19:44:07.572140 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.572030 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9829fc4e-8496-46e1-9861-05365842c043" containerName="extract" Jul 15 19:44:07.572140 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.572046 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9829fc4e-8496-46e1-9861-05365842c043" containerName="extract" Jul 15 19:44:07.572140 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.572062 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9829fc4e-8496-46e1-9861-05365842c043" containerName="util" Jul 15 19:44:07.572140 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.572070 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9829fc4e-8496-46e1-9861-05365842c043" containerName="util" Jul 15 19:44:07.572140 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.572083 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9829fc4e-8496-46e1-9861-05365842c043" containerName="pull" Jul 15 19:44:07.572140 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.572090 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9829fc4e-8496-46e1-9861-05365842c043" containerName="pull" Jul 15 19:44:07.572439 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.572162 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9829fc4e-8496-46e1-9861-05365842c043" containerName="extract" Jul 15 19:44:07.575392 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.575370 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jjtqm" Jul 15 19:44:07.577907 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.577882 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 15 19:44:07.577907 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.577901 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:44:07.578056 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.577924 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8lw6t\"" Jul 15 19:44:07.591990 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.591958 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jjtqm"] Jul 15 19:44:07.757957 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.757923 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngxbg\" (UniqueName: \"kubernetes.io/projected/99f15014-30af-4ea7-b4c7-c7ff55441033-kube-api-access-ngxbg\") pod \"cert-manager-operator-controller-manager-55c444bf86-jjtqm\" (UID: \"99f15014-30af-4ea7-b4c7-c7ff55441033\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jjtqm" Jul 15 19:44:07.758102 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.757992 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99f15014-30af-4ea7-b4c7-c7ff55441033-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jjtqm\" (UID: \"99f15014-30af-4ea7-b4c7-c7ff55441033\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jjtqm" Jul 15 19:44:07.858981 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.858867 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99f15014-30af-4ea7-b4c7-c7ff55441033-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jjtqm\" (UID: \"99f15014-30af-4ea7-b4c7-c7ff55441033\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jjtqm" Jul 15 19:44:07.858981 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.858945 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngxbg\" (UniqueName: \"kubernetes.io/projected/99f15014-30af-4ea7-b4c7-c7ff55441033-kube-api-access-ngxbg\") pod \"cert-manager-operator-controller-manager-55c444bf86-jjtqm\" (UID: \"99f15014-30af-4ea7-b4c7-c7ff55441033\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jjtqm" Jul 15 19:44:07.859287 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.859266 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99f15014-30af-4ea7-b4c7-c7ff55441033-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jjtqm\" (UID: \"99f15014-30af-4ea7-b4c7-c7ff55441033\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jjtqm" Jul 15 19:44:07.867769 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.867735 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngxbg\" (UniqueName: \"kubernetes.io/projected/99f15014-30af-4ea7-b4c7-c7ff55441033-kube-api-access-ngxbg\") pod \"cert-manager-operator-controller-manager-55c444bf86-jjtqm\" (UID: \"99f15014-30af-4ea7-b4c7-c7ff55441033\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jjtqm" Jul 15 19:44:07.884666 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:07.884636 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jjtqm" Jul 15 19:44:08.023957 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:08.023923 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jjtqm"] Jul 15 19:44:08.027539 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:44:08.027509 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f15014_30af_4ea7_b4c7_c7ff55441033.slice/crio-67d78cfeefaa6cb2553c806ed67294cf680e6dccdbdd4da1c5f10031a11a542c WatchSource:0}: Error finding container 67d78cfeefaa6cb2553c806ed67294cf680e6dccdbdd4da1c5f10031a11a542c: Status 404 returned error can't find the container with id 67d78cfeefaa6cb2553c806ed67294cf680e6dccdbdd4da1c5f10031a11a542c Jul 15 19:44:08.750668 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:08.750625 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jjtqm" event={"ID":"99f15014-30af-4ea7-b4c7-c7ff55441033","Type":"ContainerStarted","Data":"67d78cfeefaa6cb2553c806ed67294cf680e6dccdbdd4da1c5f10031a11a542c"} Jul 15 19:44:09.256011 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:09.255962 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:09.354276 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:09.354219 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:09.756966 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:09.756925 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jjtqm" event={"ID":"99f15014-30af-4ea7-b4c7-c7ff55441033","Type":"ContainerStarted","Data":"5da609bcaba16c66af6e5cb1b983df6d6e3cdf36e042c4da23c5a800b2ae02a1"} Jul 15 19:44:09.784002 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:09.783951 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jjtqm" podStartSLOduration=1.44297348 podStartE2EDuration="2.783936003s" totalImagesPullingTime="1.340962523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:44:09.781438317 +0000 UTC m=+364.025944781" watchObservedRunningTime="2026-07-15 19:44:09.783936003 +0000 UTC m=+364.028442467" Jul 15 19:44:10.149225 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:10.149136 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:10.301499 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:10.301460 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:11.046142 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.046106 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:11.083369 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.083332 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl"] Jul 15 19:44:11.086728 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.086703 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl" Jul 15 19:44:11.089786 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.089765 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 19:44:11.089993 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.089979 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xw4wp\"" Jul 15 19:44:11.090870 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.090851 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 19:44:11.098867 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.098847 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl"] Jul 15 19:44:11.140927 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.140894 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:11.186967 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.186933 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79aa072a-00ce-407b-ad4a-b11976a987db-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl\" (UID: \"79aa072a-00ce-407b-ad4a-b11976a987db\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl" Jul 15 19:44:11.187127 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.186980 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79aa072a-00ce-407b-ad4a-b11976a987db-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl\" (UID: \"79aa072a-00ce-407b-ad4a-b11976a987db\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl" Jul 15 19:44:11.187127 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.187058 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwppb\" (UniqueName: \"kubernetes.io/projected/79aa072a-00ce-407b-ad4a-b11976a987db-kube-api-access-wwppb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl\" (UID: \"79aa072a-00ce-407b-ad4a-b11976a987db\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl" Jul 15 19:44:11.287970 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.287934 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79aa072a-00ce-407b-ad4a-b11976a987db-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl\" (UID: \"79aa072a-00ce-407b-ad4a-b11976a987db\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl" Jul 15 19:44:11.288106 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.287987 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79aa072a-00ce-407b-ad4a-b11976a987db-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl\" (UID: \"79aa072a-00ce-407b-ad4a-b11976a987db\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl" Jul 15 19:44:11.288106 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.288054 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwppb\" (UniqueName: \"kubernetes.io/projected/79aa072a-00ce-407b-ad4a-b11976a987db-kube-api-access-wwppb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl\" (UID: \"79aa072a-00ce-407b-ad4a-b11976a987db\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl" Jul 15 19:44:11.288367 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.288347 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79aa072a-00ce-407b-ad4a-b11976a987db-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl\" (UID: \"79aa072a-00ce-407b-ad4a-b11976a987db\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl" Jul 15 19:44:11.288400 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.288368 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79aa072a-00ce-407b-ad4a-b11976a987db-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl\" (UID: \"79aa072a-00ce-407b-ad4a-b11976a987db\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl" Jul 15 19:44:11.297983 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.297921 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwppb\" (UniqueName: \"kubernetes.io/projected/79aa072a-00ce-407b-ad4a-b11976a987db-kube-api-access-wwppb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl\" (UID: \"79aa072a-00ce-407b-ad4a-b11976a987db\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl" Jul 15 19:44:11.396180 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.396140 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl" Jul 15 19:44:11.532665 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.532640 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl"] Jul 15 19:44:11.535207 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:44:11.535179 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79aa072a_00ce_407b_ad4a_b11976a987db.slice/crio-c739dee80a15d79b76b1932a588ae06984baacdf368c7367d833f6ec7341d978 WatchSource:0}: Error finding container c739dee80a15d79b76b1932a588ae06984baacdf368c7367d833f6ec7341d978: Status 404 returned error can't find the container with id c739dee80a15d79b76b1932a588ae06984baacdf368c7367d833f6ec7341d978 Jul 15 19:44:11.766746 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.766712 2554 generic.go:356] "Generic (PLEG): container finished" podID="79aa072a-00ce-407b-ad4a-b11976a987db" containerID="d96b9e3c63604af143853bc843062a94f3ed549882b91240e5674afe65d34981" exitCode=0 Jul 15 19:44:11.766940 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.766797 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl" event={"ID":"79aa072a-00ce-407b-ad4a-b11976a987db","Type":"ContainerDied","Data":"d96b9e3c63604af143853bc843062a94f3ed549882b91240e5674afe65d34981"} Jul 15 19:44:11.766940 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:11.766847 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl" event={"ID":"79aa072a-00ce-407b-ad4a-b11976a987db","Type":"ContainerStarted","Data":"c739dee80a15d79b76b1932a588ae06984baacdf368c7367d833f6ec7341d978"} Jul 15 19:44:13.249682 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:13.249641 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:13.451121 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:13.451078 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:13.776981 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:13.776951 2554 generic.go:356] "Generic (PLEG): container finished" podID="79aa072a-00ce-407b-ad4a-b11976a987db" containerID="7852e504729859da32d37fd2f04e18edeac3918fcd93e4eb843e1cff2074d9ab" exitCode=0 Jul 15 19:44:13.777165 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:13.777034 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl" event={"ID":"79aa072a-00ce-407b-ad4a-b11976a987db","Type":"ContainerDied","Data":"7852e504729859da32d37fd2f04e18edeac3918fcd93e4eb843e1cff2074d9ab"} Jul 15 19:44:14.783639 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:14.783607 2554 generic.go:356] "Generic (PLEG): container finished" podID="79aa072a-00ce-407b-ad4a-b11976a987db" containerID="c0a000424f6d22605a5848b5f34b8314bc624afa7bc2d5479543081bd14329f2" exitCode=0 Jul 15 19:44:14.784018 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:14.783661 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl" event={"ID":"79aa072a-00ce-407b-ad4a-b11976a987db","Type":"ContainerDied","Data":"c0a000424f6d22605a5848b5f34b8314bc624afa7bc2d5479543081bd14329f2"} Jul 15 19:44:15.912659 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:15.912622 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl" Jul 15 19:44:15.925319 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:15.925287 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79aa072a-00ce-407b-ad4a-b11976a987db-util\") pod \"79aa072a-00ce-407b-ad4a-b11976a987db\" (UID: \"79aa072a-00ce-407b-ad4a-b11976a987db\") " Jul 15 19:44:15.925487 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:15.925350 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwppb\" (UniqueName: \"kubernetes.io/projected/79aa072a-00ce-407b-ad4a-b11976a987db-kube-api-access-wwppb\") pod \"79aa072a-00ce-407b-ad4a-b11976a987db\" (UID: \"79aa072a-00ce-407b-ad4a-b11976a987db\") " Jul 15 19:44:15.925487 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:15.925384 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79aa072a-00ce-407b-ad4a-b11976a987db-bundle\") pod \"79aa072a-00ce-407b-ad4a-b11976a987db\" (UID: \"79aa072a-00ce-407b-ad4a-b11976a987db\") " Jul 15 19:44:15.925878 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:15.925838 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79aa072a-00ce-407b-ad4a-b11976a987db-bundle" (OuterVolumeSpecName: "bundle") pod "79aa072a-00ce-407b-ad4a-b11976a987db" (UID: "79aa072a-00ce-407b-ad4a-b11976a987db"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:44:15.927718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:15.927693 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79aa072a-00ce-407b-ad4a-b11976a987db-kube-api-access-wwppb" (OuterVolumeSpecName: "kube-api-access-wwppb") pod "79aa072a-00ce-407b-ad4a-b11976a987db" (UID: "79aa072a-00ce-407b-ad4a-b11976a987db"). InnerVolumeSpecName "kube-api-access-wwppb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:44:15.931473 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:15.931444 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79aa072a-00ce-407b-ad4a-b11976a987db-util" (OuterVolumeSpecName: "util") pod "79aa072a-00ce-407b-ad4a-b11976a987db" (UID: "79aa072a-00ce-407b-ad4a-b11976a987db"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:44:16.017907 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:16.017863 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:16.025902 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:16.025874 2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79aa072a-00ce-407b-ad4a-b11976a987db-bundle\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:44:16.025902 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:16.025899 2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79aa072a-00ce-407b-ad4a-b11976a987db-util\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:44:16.025902 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:16.025908 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwppb\" (UniqueName: \"kubernetes.io/projected/79aa072a-00ce-407b-ad4a-b11976a987db-kube-api-access-wwppb\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:44:16.083236 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:16.083145 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:16.793625 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:16.793587 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl" event={"ID":"79aa072a-00ce-407b-ad4a-b11976a987db","Type":"ContainerDied","Data":"c739dee80a15d79b76b1932a588ae06984baacdf368c7367d833f6ec7341d978"} Jul 15 19:44:16.793625 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:16.793611 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fht7vl" Jul 15 19:44:16.793625 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:16.793632 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c739dee80a15d79b76b1932a588ae06984baacdf368c7367d833f6ec7341d978" Jul 15 19:44:19.352728 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.352694 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wvhdd"] Jul 15 19:44:19.353100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.352994 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79aa072a-00ce-407b-ad4a-b11976a987db" containerName="util" Jul 15 19:44:19.353100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.353005 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="79aa072a-00ce-407b-ad4a-b11976a987db" containerName="util" Jul 15 19:44:19.353100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.353020 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79aa072a-00ce-407b-ad4a-b11976a987db" containerName="pull" Jul 15 19:44:19.353100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.353025 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="79aa072a-00ce-407b-ad4a-b11976a987db" containerName="pull" Jul 15 19:44:19.353100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.353036 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79aa072a-00ce-407b-ad4a-b11976a987db" containerName="extract" Jul 15 19:44:19.353100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.353042 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="79aa072a-00ce-407b-ad4a-b11976a987db" containerName="extract" Jul 15 19:44:19.353100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.353087 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="79aa072a-00ce-407b-ad4a-b11976a987db" containerName="extract" Jul 15 19:44:19.354921 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.354901 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wvhdd" Jul 15 19:44:19.357964 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.357939 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-njzrk\"" Jul 15 19:44:19.358736 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.358719 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 15 19:44:19.358736 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.358729 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 15 19:44:19.366724 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.366701 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wvhdd"] Jul 15 19:44:19.447310 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.447272 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37781012-ef5d-4f90-bf4c-def1b1dd7b96-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wvhdd\" (UID: \"37781012-ef5d-4f90-bf4c-def1b1dd7b96\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wvhdd" Jul 15 19:44:19.447459 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.447326 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwdhl\" (UniqueName: \"kubernetes.io/projected/37781012-ef5d-4f90-bf4c-def1b1dd7b96-kube-api-access-lwdhl\") pod \"cert-manager-webhook-6486ffd997-wvhdd\" (UID: \"37781012-ef5d-4f90-bf4c-def1b1dd7b96\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wvhdd" Jul 15 19:44:19.547731 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.547690 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37781012-ef5d-4f90-bf4c-def1b1dd7b96-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wvhdd\" (UID: \"37781012-ef5d-4f90-bf4c-def1b1dd7b96\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wvhdd" Jul 15 19:44:19.547731 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.547739 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwdhl\" (UniqueName: \"kubernetes.io/projected/37781012-ef5d-4f90-bf4c-def1b1dd7b96-kube-api-access-lwdhl\") pod \"cert-manager-webhook-6486ffd997-wvhdd\" (UID: \"37781012-ef5d-4f90-bf4c-def1b1dd7b96\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wvhdd" Jul 15 19:44:19.556717 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.556680 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37781012-ef5d-4f90-bf4c-def1b1dd7b96-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wvhdd\" (UID: \"37781012-ef5d-4f90-bf4c-def1b1dd7b96\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wvhdd" Jul 15 19:44:19.556932 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.556909 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwdhl\" (UniqueName: \"kubernetes.io/projected/37781012-ef5d-4f90-bf4c-def1b1dd7b96-kube-api-access-lwdhl\") pod \"cert-manager-webhook-6486ffd997-wvhdd\" (UID: \"37781012-ef5d-4f90-bf4c-def1b1dd7b96\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wvhdd" Jul 15 19:44:19.673124 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.673082 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wvhdd" Jul 15 19:44:19.804900 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:19.804840 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wvhdd"] Jul 15 19:44:19.807787 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:44:19.807738 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37781012_ef5d_4f90_bf4c_def1b1dd7b96.slice/crio-cd43fd66913021a47b8508574ecb1c928cf3ab12f6d7c9fa5ddb7f798d8d6fe1 WatchSource:0}: Error finding container cd43fd66913021a47b8508574ecb1c928cf3ab12f6d7c9fa5ddb7f798d8d6fe1: Status 404 returned error can't find the container with id cd43fd66913021a47b8508574ecb1c928cf3ab12f6d7c9fa5ddb7f798d8d6fe1 Jul 15 19:44:20.556324 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:20.556270 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:20.819223 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:20.819136 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wvhdd" event={"ID":"37781012-ef5d-4f90-bf4c-def1b1dd7b96","Type":"ContainerStarted","Data":"cd43fd66913021a47b8508574ecb1c928cf3ab12f6d7c9fa5ddb7f798d8d6fe1"} Jul 15 19:44:21.030608 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:21.030562 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:21.129272 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:21.129183 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:21.447700 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:21.447658 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:22.566776 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:22.566678 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:22.667146 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:22.667106 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:22.830799 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:22.830680 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wvhdd" event={"ID":"37781012-ef5d-4f90-bf4c-def1b1dd7b96","Type":"ContainerStarted","Data":"86fe6a2ad60720a6cac6c53f0b8aaa0e3cc312fe9f0c1c2fd5ca085a71cf0ea8"} Jul 15 19:44:22.830799 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:22.830726 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wvhdd" Jul 15 19:44:22.859250 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:22.859200 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-wvhdd" podStartSLOduration=1.368077818 podStartE2EDuration="3.859183972s" totalImagesPullingTime="2.491106154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:44:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:44:22.857900611 +0000 UTC m=+377.102407073" watchObservedRunningTime="2026-07-15 19:44:22.859183972 +0000 UTC m=+377.103690434" Jul 15 19:44:23.650904 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:23.650870 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kdxdb"] Jul 15 19:44:23.656407 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:23.656387 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kdxdb" Jul 15 19:44:23.659060 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:23.659035 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:44:23.659060 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:23.659052 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 19:44:23.659867 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:23.659852 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-4jhwb\"" Jul 15 19:44:23.663272 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:23.663122 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kdxdb"] Jul 15 19:44:23.671257 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:23.671232 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhdd5\" (UniqueName: \"kubernetes.io/projected/2d0a0425-17b3-4b5c-8cf1-a73b0ead7507-kube-api-access-dhdd5\") pod \"openshift-lws-operator-fd8ccff4c-kdxdb\" (UID: \"2d0a0425-17b3-4b5c-8cf1-a73b0ead7507\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kdxdb" Jul 15 19:44:23.671386 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:23.671266 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d0a0425-17b3-4b5c-8cf1-a73b0ead7507-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kdxdb\" (UID: \"2d0a0425-17b3-4b5c-8cf1-a73b0ead7507\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kdxdb" Jul 15 19:44:23.771686 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:23.771651 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhdd5\" (UniqueName: \"kubernetes.io/projected/2d0a0425-17b3-4b5c-8cf1-a73b0ead7507-kube-api-access-dhdd5\") pod \"openshift-lws-operator-fd8ccff4c-kdxdb\" (UID: \"2d0a0425-17b3-4b5c-8cf1-a73b0ead7507\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kdxdb" Jul 15 19:44:23.771686 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:23.771686 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d0a0425-17b3-4b5c-8cf1-a73b0ead7507-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kdxdb\" (UID: \"2d0a0425-17b3-4b5c-8cf1-a73b0ead7507\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kdxdb" Jul 15 19:44:23.772053 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:23.772034 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d0a0425-17b3-4b5c-8cf1-a73b0ead7507-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kdxdb\" (UID: \"2d0a0425-17b3-4b5c-8cf1-a73b0ead7507\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kdxdb" Jul 15 19:44:23.781207 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:23.781172 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhdd5\" (UniqueName: \"kubernetes.io/projected/2d0a0425-17b3-4b5c-8cf1-a73b0ead7507-kube-api-access-dhdd5\") pod \"openshift-lws-operator-fd8ccff4c-kdxdb\" (UID: \"2d0a0425-17b3-4b5c-8cf1-a73b0ead7507\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kdxdb" Jul 15 19:44:23.966022 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:23.965984 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kdxdb" Jul 15 19:44:24.104512 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:24.104480 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kdxdb"] Jul 15 19:44:24.106619 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:44:24.106593 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d0a0425_17b3_4b5c_8cf1_a73b0ead7507.slice/crio-2cb1995f78a1776899de84224e52c6d7d74c5eb7f500f8557f0f0102dee8c45d WatchSource:0}: Error finding container 2cb1995f78a1776899de84224e52c6d7d74c5eb7f500f8557f0f0102dee8c45d: Status 404 returned error can't find the container with id 2cb1995f78a1776899de84224e52c6d7d74c5eb7f500f8557f0f0102dee8c45d Jul 15 19:44:24.841350 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:24.841315 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kdxdb" event={"ID":"2d0a0425-17b3-4b5c-8cf1-a73b0ead7507","Type":"ContainerStarted","Data":"2cb1995f78a1776899de84224e52c6d7d74c5eb7f500f8557f0f0102dee8c45d"} Jul 15 19:44:27.857115 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:27.857079 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kdxdb" event={"ID":"2d0a0425-17b3-4b5c-8cf1-a73b0ead7507","Type":"ContainerStarted","Data":"3a78c2244a8cff8959125401d58c4abb57bb3f61fe783814d78098d3b9d08d89"} Jul 15 19:44:27.875877 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:27.875828 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kdxdb" podStartSLOduration=1.924461953 podStartE2EDuration="4.875812644s" totalImagesPullingTime="2.951350691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:44:27.875736288 +0000 UTC m=+382.120242750" watchObservedRunningTime="2026-07-15 19:44:27.875812644 +0000 UTC m=+382.120319104" Jul 15 19:44:28.558890 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:28.558854 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:28.664843 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:28.664785 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:28.837813 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:28.837710 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wvhdd" Jul 15 19:44:30.304919 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:30.304878 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:31.324016 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:31.323986 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8"] Jul 15 19:44:31.339936 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:31.339908 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8"] Jul 15 19:44:31.340095 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:31.340017 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8" Jul 15 19:44:31.342697 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:31.342669 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 19:44:31.342850 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:31.342674 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 19:44:31.343709 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:31.343688 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xw4wp\"" Jul 15 19:44:31.433347 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:31.433310 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8\" (UID: \"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8" Jul 15 19:44:31.433524 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:31.433361 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hktb8\" (UniqueName: \"kubernetes.io/projected/05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5-kube-api-access-hktb8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8\" (UID: \"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8" Jul 15 19:44:31.433524 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:31.433436 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8\" (UID: \"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8" Jul 15 19:44:31.534472 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:31.534434 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hktb8\" (UniqueName: \"kubernetes.io/projected/05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5-kube-api-access-hktb8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8\" (UID: \"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8" Jul 15 19:44:31.534646 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:31.534493 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8\" (UID: \"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8" Jul 15 19:44:31.534646 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:31.534538 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8\" (UID: \"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8" Jul 15 19:44:31.534944 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:31.534922 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8\" (UID: \"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8" Jul 15 19:44:31.534999 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:31.534982 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8\" (UID: \"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8" Jul 15 19:44:31.543424 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:31.543406 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hktb8\" (UniqueName: \"kubernetes.io/projected/05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5-kube-api-access-hktb8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8\" (UID: \"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8" Jul 15 19:44:31.649517 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:31.649426 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8" Jul 15 19:44:31.804857 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:31.804789 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8"] Jul 15 19:44:31.807553 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:44:31.807530 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ec98ff_93a6_4e2a_a7f3_b188ad7cbba5.slice/crio-1fd4523e1666b6b786ed56b55627aabb029ce2b407cafbb4c594f627708cdb8a WatchSource:0}: Error finding container 1fd4523e1666b6b786ed56b55627aabb029ce2b407cafbb4c594f627708cdb8a: Status 404 returned error can't find the container with id 1fd4523e1666b6b786ed56b55627aabb029ce2b407cafbb4c594f627708cdb8a Jul 15 19:44:31.874804 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:31.874776 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8" event={"ID":"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5","Type":"ContainerStarted","Data":"1fd4523e1666b6b786ed56b55627aabb029ce2b407cafbb4c594f627708cdb8a"} Jul 15 19:44:32.880801 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:32.880693 2554 generic.go:356] "Generic (PLEG): container finished" podID="05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5" containerID="1b1e567968e34b31abcacc31bf22703afa0dad0b746c55dcbe2d9bbcbb3baeff" exitCode=0 Jul 15 19:44:32.880801 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:32.880785 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8" event={"ID":"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5","Type":"ContainerDied","Data":"1b1e567968e34b31abcacc31bf22703afa0dad0b746c55dcbe2d9bbcbb3baeff"} Jul 15 19:44:33.887388 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:33.887354 2554 generic.go:356] "Generic (PLEG): container finished" podID="05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5" containerID="c9def77506af5867d9d5edd9cf1b3ac31de0e8576e5130631b3318033a3e280c" exitCode=0 Jul 15 19:44:33.887787 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:33.887435 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8" event={"ID":"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5","Type":"ContainerDied","Data":"c9def77506af5867d9d5edd9cf1b3ac31de0e8576e5130631b3318033a3e280c"} Jul 15 19:44:34.894083 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:34.894046 2554 generic.go:356] "Generic (PLEG): container finished" podID="05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5" containerID="c446ac01e75458a0aa87e5d1ff601c19f16e4084e37257b92e8446fb77621ad0" exitCode=0 Jul 15 19:44:34.894481 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:34.894136 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8" event={"ID":"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5","Type":"ContainerDied","Data":"c446ac01e75458a0aa87e5d1ff601c19f16e4084e37257b92e8446fb77621ad0"} Jul 15 19:44:36.021389 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:36.021364 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8" Jul 15 19:44:36.073833 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:36.073804 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5-bundle\") pod \"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5\" (UID: \"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5\") " Jul 15 19:44:36.073979 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:36.073880 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5-util\") pod \"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5\" (UID: \"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5\") " Jul 15 19:44:36.073979 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:36.073929 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hktb8\" (UniqueName: \"kubernetes.io/projected/05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5-kube-api-access-hktb8\") pod \"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5\" (UID: \"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5\") " Jul 15 19:44:36.074561 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:36.074528 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5-bundle" (OuterVolumeSpecName: "bundle") pod "05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5" (UID: "05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:44:36.076140 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:36.076118 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5-kube-api-access-hktb8" (OuterVolumeSpecName: "kube-api-access-hktb8") pod "05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5" (UID: "05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5"). InnerVolumeSpecName "kube-api-access-hktb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:44:36.079438 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:36.079404 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5-util" (OuterVolumeSpecName: "util") pod "05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5" (UID: "05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:44:36.174668 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:36.174640 2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5-bundle\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:44:36.174668 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:36.174665 2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5-util\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:44:36.174880 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:36.174677 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hktb8\" (UniqueName: \"kubernetes.io/projected/05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5-kube-api-access-hktb8\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:44:36.904702 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:36.904663 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8" event={"ID":"05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5","Type":"ContainerDied","Data":"1fd4523e1666b6b786ed56b55627aabb029ce2b407cafbb4c594f627708cdb8a"} Jul 15 19:44:36.904702 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:36.904689 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzljx8" Jul 15 19:44:36.905261 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:36.904715 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fd4523e1666b6b786ed56b55627aabb029ce2b407cafbb4c594f627708cdb8a" Jul 15 19:44:42.421689 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.421643 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:42.536711 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.536627 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:42.652553 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.652516 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:42.770481 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.770397 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48"] Jul 15 19:44:42.770715 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.770701 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5" containerName="util" Jul 15 19:44:42.770715 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.770715 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5" containerName="util" Jul 15 19:44:42.770867 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.770724 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5" containerName="extract" Jul 15 19:44:42.770867 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.770729 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5" containerName="extract" Jul 15 19:44:42.770867 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.770735 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5" containerName="pull" Jul 15 19:44:42.770867 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.770741 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5" containerName="pull" Jul 15 19:44:42.770867 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.770822 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="05ec98ff-93a6-4e2a-a7f3-b188ad7cbba5" containerName="extract" Jul 15 19:44:42.773857 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.773841 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48" Jul 15 19:44:42.778283 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.778255 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-x7cc7\"" Jul 15 19:44:42.778420 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.778283 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 15 19:44:42.778683 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.778666 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 15 19:44:42.779181 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.779169 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 15 19:44:42.789978 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.789951 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48"] Jul 15 19:44:42.810460 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.810426 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:42.820555 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.820531 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93279220-48fc-4ec2-b2ac-14b2d6abf2a6-cert\") pod \"lws-controller-manager-7c89ffd76c-hxx48\" (UID: \"93279220-48fc-4ec2-b2ac-14b2d6abf2a6\") " pod="openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48" Jul 15 19:44:42.820685 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.820577 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/93279220-48fc-4ec2-b2ac-14b2d6abf2a6-metrics-cert\") pod \"lws-controller-manager-7c89ffd76c-hxx48\" (UID: \"93279220-48fc-4ec2-b2ac-14b2d6abf2a6\") " pod="openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48" Jul 15 19:44:42.820685 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.820601 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdp7j\" (UniqueName: \"kubernetes.io/projected/93279220-48fc-4ec2-b2ac-14b2d6abf2a6-kube-api-access-kdp7j\") pod \"lws-controller-manager-7c89ffd76c-hxx48\" (UID: \"93279220-48fc-4ec2-b2ac-14b2d6abf2a6\") " pod="openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48" Jul 15 19:44:42.820685 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.820668 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/93279220-48fc-4ec2-b2ac-14b2d6abf2a6-manager-config\") pod \"lws-controller-manager-7c89ffd76c-hxx48\" (UID: \"93279220-48fc-4ec2-b2ac-14b2d6abf2a6\") " pod="openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48" Jul 15 19:44:42.921827 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.921789 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/93279220-48fc-4ec2-b2ac-14b2d6abf2a6-metrics-cert\") pod \"lws-controller-manager-7c89ffd76c-hxx48\" (UID: \"93279220-48fc-4ec2-b2ac-14b2d6abf2a6\") " pod="openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48" Jul 15 19:44:42.921827 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.921831 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdp7j\" (UniqueName: \"kubernetes.io/projected/93279220-48fc-4ec2-b2ac-14b2d6abf2a6-kube-api-access-kdp7j\") pod \"lws-controller-manager-7c89ffd76c-hxx48\" (UID: \"93279220-48fc-4ec2-b2ac-14b2d6abf2a6\") " pod="openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48" Jul 15 19:44:42.922027 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.921872 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/93279220-48fc-4ec2-b2ac-14b2d6abf2a6-manager-config\") pod \"lws-controller-manager-7c89ffd76c-hxx48\" (UID: \"93279220-48fc-4ec2-b2ac-14b2d6abf2a6\") " pod="openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48" Jul 15 19:44:42.922027 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.921898 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93279220-48fc-4ec2-b2ac-14b2d6abf2a6-cert\") pod \"lws-controller-manager-7c89ffd76c-hxx48\" (UID: \"93279220-48fc-4ec2-b2ac-14b2d6abf2a6\") " pod="openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48" Jul 15 19:44:42.922640 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.922616 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/93279220-48fc-4ec2-b2ac-14b2d6abf2a6-manager-config\") pod \"lws-controller-manager-7c89ffd76c-hxx48\" (UID: \"93279220-48fc-4ec2-b2ac-14b2d6abf2a6\") " pod="openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48" Jul 15 19:44:42.924424 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.924394 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93279220-48fc-4ec2-b2ac-14b2d6abf2a6-cert\") pod \"lws-controller-manager-7c89ffd76c-hxx48\" (UID: \"93279220-48fc-4ec2-b2ac-14b2d6abf2a6\") " pod="openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48" Jul 15 19:44:42.924539 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.924507 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/93279220-48fc-4ec2-b2ac-14b2d6abf2a6-metrics-cert\") pod \"lws-controller-manager-7c89ffd76c-hxx48\" (UID: \"93279220-48fc-4ec2-b2ac-14b2d6abf2a6\") " pod="openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48" Jul 15 19:44:42.931429 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:42.931394 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdp7j\" (UniqueName: \"kubernetes.io/projected/93279220-48fc-4ec2-b2ac-14b2d6abf2a6-kube-api-access-kdp7j\") pod \"lws-controller-manager-7c89ffd76c-hxx48\" (UID: \"93279220-48fc-4ec2-b2ac-14b2d6abf2a6\") " pod="openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48" Jul 15 19:44:43.056020 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:43.055926 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:43.084078 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:43.084046 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48" Jul 15 19:44:43.233460 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:43.233436 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48"] Jul 15 19:44:43.235962 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:44:43.235927 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93279220_48fc_4ec2_b2ac_14b2d6abf2a6.slice/crio-2a80f7f29092f74b28613fd63e5150d24b5b6fbd1f08520bba4c72e1aeffb616 WatchSource:0}: Error finding container 2a80f7f29092f74b28613fd63e5150d24b5b6fbd1f08520bba4c72e1aeffb616: Status 404 returned error can't find the container with id 2a80f7f29092f74b28613fd63e5150d24b5b6fbd1f08520bba4c72e1aeffb616 Jul 15 19:44:43.566027 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:43.565993 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:43.677535 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:43.677496 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:43.819867 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:43.819786 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s"] Jul 15 19:44:43.826224 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:43.826202 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" Jul 15 19:44:43.829001 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:43.828977 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 19:44:43.829781 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:43.829761 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 19:44:43.829882 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:43.829842 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8svbl\"" Jul 15 19:44:43.830027 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:43.830015 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 19:44:43.830065 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:43.830042 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 19:44:43.859846 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:43.859820 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s"] Jul 15 19:44:43.940442 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:43.940403 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48" event={"ID":"93279220-48fc-4ec2-b2ac-14b2d6abf2a6","Type":"ContainerStarted","Data":"2a80f7f29092f74b28613fd63e5150d24b5b6fbd1f08520bba4c72e1aeffb616"} Jul 15 19:44:43.942867 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:43.942842 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c8f5fdd94-bh52s\" (UID: \"ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c\") " pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" Jul 15 19:44:43.942989 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:43.942879 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c8f5fdd94-bh52s\" (UID: \"ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c\") " pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" Jul 15 19:44:43.942989 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:43.942950 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c-tmp\") pod \"opendatahub-operator-controller-manager-6c8f5fdd94-bh52s\" (UID: \"ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c\") " pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" Jul 15 19:44:43.942989 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:43.942979 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nllq5\" (UniqueName: \"kubernetes.io/projected/ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c-kube-api-access-nllq5\") pod \"opendatahub-operator-controller-manager-6c8f5fdd94-bh52s\" (UID: \"ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c\") " pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" Jul 15 19:44:43.943132 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:43.943067 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c-manifests\") pod \"opendatahub-operator-controller-manager-6c8f5fdd94-bh52s\" (UID: \"ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c\") " pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" Jul 15 19:44:44.044277 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:44.044225 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c-manifests\") pod \"opendatahub-operator-controller-manager-6c8f5fdd94-bh52s\" (UID: \"ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c\") " pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" Jul 15 19:44:44.044463 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:44.044286 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c8f5fdd94-bh52s\" (UID: \"ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c\") " pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" Jul 15 19:44:44.044463 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:44.044315 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c8f5fdd94-bh52s\" (UID: \"ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c\") " pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" Jul 15 19:44:44.044463 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:44.044365 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c-tmp\") pod \"opendatahub-operator-controller-manager-6c8f5fdd94-bh52s\" (UID: \"ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c\") " pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" Jul 15 19:44:44.044463 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:44.044403 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nllq5\" (UniqueName: \"kubernetes.io/projected/ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c-kube-api-access-nllq5\") pod \"opendatahub-operator-controller-manager-6c8f5fdd94-bh52s\" (UID: \"ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c\") " pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" Jul 15 19:44:44.044938 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:44.044910 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c-manifests\") pod \"opendatahub-operator-controller-manager-6c8f5fdd94-bh52s\" (UID: \"ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c\") " pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" Jul 15 19:44:44.045843 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:44.045768 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:44.047348 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:44.047323 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c-tmp\") pod \"opendatahub-operator-controller-manager-6c8f5fdd94-bh52s\" (UID: \"ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c\") " pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" Jul 15 19:44:44.047453 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:44.047401 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c8f5fdd94-bh52s\" (UID: \"ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c\") " pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" Jul 15 19:44:44.047610 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:44.047588 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c8f5fdd94-bh52s\" (UID: \"ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c\") " pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" Jul 15 19:44:44.054407 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:44.054384 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nllq5\" (UniqueName: \"kubernetes.io/projected/ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c-kube-api-access-nllq5\") pod \"opendatahub-operator-controller-manager-6c8f5fdd94-bh52s\" (UID: \"ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c\") " pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" Jul 15 19:44:44.137679 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:44.137589 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" Jul 15 19:44:44.145274 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:44.145212 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:44.246620 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:44.246484 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:44.313410 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:44.313380 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s"] Jul 15 19:44:44.316194 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:44:44.316167 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc9f9e0_bb94_40f7_91de_2a91d5ed4e4c.slice/crio-be2d39ec7ad31ececcc8de234d347e53cca06d62c494078ac7a70c5c1dd162b4 WatchSource:0}: Error finding container be2d39ec7ad31ececcc8de234d347e53cca06d62c494078ac7a70c5c1dd162b4: Status 404 returned error can't find the container with id be2d39ec7ad31ececcc8de234d347e53cca06d62c494078ac7a70c5c1dd162b4 Jul 15 19:44:44.346049 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:44.346009 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:44.454227 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:44.454186 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:44.555799 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:44.555717 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:44.947423 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:44.947386 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" event={"ID":"ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c","Type":"ContainerStarted","Data":"be2d39ec7ad31ececcc8de234d347e53cca06d62c494078ac7a70c5c1dd162b4"} Jul 15 19:44:45.954744 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:45.954694 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48" event={"ID":"93279220-48fc-4ec2-b2ac-14b2d6abf2a6","Type":"ContainerStarted","Data":"60fa653758e24e8d6be9ac0c98134c46e234ddbe4eae3601ba8e9d9af33fa8b2"} Jul 15 19:44:45.955197 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:45.954833 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48" Jul 15 19:44:45.978687 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:45.978627 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48" podStartSLOduration=2.310644583 podStartE2EDuration="3.978611054s" totalImagesPullingTime="1.667966471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:44:45.977990487 +0000 UTC m=+400.222496949" watchObservedRunningTime="2026-07-15 19:44:45.978611054 +0000 UTC m=+400.223117582" Jul 15 19:44:47.965933 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:47.965897 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" event={"ID":"ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c","Type":"ContainerStarted","Data":"6582869baec8f6a9581f6132a4abe529ac5d3f652aa2dbba11c12c9e9a7e10f4"} Jul 15 19:44:48.972506 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:48.972474 2554 generic.go:356] "Generic (PLEG): container finished" podID="ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c" containerID="6582869baec8f6a9581f6132a4abe529ac5d3f652aa2dbba11c12c9e9a7e10f4" exitCode=0 Jul 15 19:44:48.972949 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:48.972569 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" event={"ID":"ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c","Type":"ContainerDied","Data":"6582869baec8f6a9581f6132a4abe529ac5d3f652aa2dbba11c12c9e9a7e10f4"} Jul 15 19:44:49.978476 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:49.978434 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" event={"ID":"ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c","Type":"ContainerStarted","Data":"ff27367ea50004f825ad86ce2dd7a6cd52b5a7f6a64efabb65a2846c10802e4b"} Jul 15 19:44:49.978970 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:49.978528 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" Jul 15 19:44:50.001351 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:50.001281 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" podStartSLOduration=2.529361256 podStartE2EDuration="7.001261945s" totalImagesPullingTime="3.40469267s" totalInitContainerRuntime="1.067208019s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:44:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:44:49.998721716 +0000 UTC m=+404.243228177" watchObservedRunningTime="2026-07-15 19:44:50.001261945 +0000 UTC m=+404.245768409" Jul 15 19:44:56.962770 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:44:56.962722 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c89ffd76c-hxx48" Jul 15 19:45:00.985155 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:00.985124 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6c8f5fdd94-bh52s" Jul 15 19:45:02.996302 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:02.996252 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:03.110591 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:03.110560 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:08.830598 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:08.830558 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:09.163593 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:09.163495 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:09.233019 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:09.232983 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:09.337081 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:09.337047 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:09.542381 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:09.542340 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:09.787961 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:09.787919 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:10.090354 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:10.090317 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:10.287332 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:10.287303 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:10.358734 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:10.358641 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:11.473020 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:11.472961 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:15.740096 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:15.740062 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:15.843155 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:15.843122 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:16.358301 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.358266 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb"] Jul 15 19:45:16.365822 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.365795 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.369200 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.369178 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 15 19:45:16.372646 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.372494 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pqcqt\"" Jul 15 19:45:16.373287 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.373271 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 15 19:45:16.373570 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.373553 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 19:45:16.373828 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.373813 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 15 19:45:16.383205 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.383180 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb"] Jul 15 19:45:16.417507 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.417476 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.417628 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.417517 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.417628 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.417587 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn8mv\" (UniqueName: \"kubernetes.io/projected/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-kube-api-access-vn8mv\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.417628 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.417615 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.417733 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.417633 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.417733 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.417658 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.417733 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.417675 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.518552 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.518519 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn8mv\" (UniqueName: \"kubernetes.io/projected/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-kube-api-access-vn8mv\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.518552 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.518560 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.518818 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.518600 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.518818 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.518634 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.518818 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.518651 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.518818 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.518680 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.519120 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.519093 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.519953 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.519889 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.521830 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.521811 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.521929 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.521905 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.521987 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.521963 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.522284 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.522261 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.538095 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.538063 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.538194 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.538103 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn8mv\" (UniqueName: \"kubernetes.io/projected/d49f4e21-2e82-48b2-ab85-9d2dae1a6241-kube-api-access-vn8mv\") pod \"istiod-openshift-gateway-94bb8fbfd-8nxrb\" (UID: \"d49f4e21-2e82-48b2-ab85-9d2dae1a6241\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.567493 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.567459 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:16.676064 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.676031 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:16.873878 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.873850 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb"] Jul 15 19:45:16.874625 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:45:16.874596 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49f4e21_2e82_48b2_ab85_9d2dae1a6241.slice/crio-892b9fbace52dc426ea2ba2bc1270b4483af9ac1a87700ea0bcbe98004a48934 WatchSource:0}: Error finding container 892b9fbace52dc426ea2ba2bc1270b4483af9ac1a87700ea0bcbe98004a48934: Status 404 returned error can't find the container with id 892b9fbace52dc426ea2ba2bc1270b4483af9ac1a87700ea0bcbe98004a48934 Jul 15 19:45:16.957147 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:16.957051 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:17.105022 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:17.104981 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" event={"ID":"d49f4e21-2e82-48b2-ab85-9d2dae1a6241","Type":"ContainerStarted","Data":"892b9fbace52dc426ea2ba2bc1270b4483af9ac1a87700ea0bcbe98004a48934"} Jul 15 19:45:17.285224 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:17.285093 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:19.704318 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:19.704153 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:19.895457 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:19.895418 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:20.085913 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:20.085838 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:20.198969 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:20.198918 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:20.472001 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:20.471951 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 19:45:20.472139 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:20.472029 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 19:45:21.129319 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:21.129279 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" event={"ID":"d49f4e21-2e82-48b2-ab85-9d2dae1a6241","Type":"ContainerStarted","Data":"77095ce553517c4cc863d1534cc7d79071e0f56133daf3aa71c7a989cf689cdb"} Jul 15 19:45:21.129812 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:21.129433 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:21.131372 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:21.131342 2554 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-8nxrb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 19:45:21.131497 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:21.131400 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" podUID="d49f4e21-2e82-48b2-ab85-9d2dae1a6241" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 19:45:21.134203 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:21.134122 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:21.161000 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:21.160947 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" podStartSLOduration=1.5660112590000002 podStartE2EDuration="5.160931051s" totalImagesPullingTime="3.594919792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:45:21.15934389 +0000 UTC m=+435.403850351" watchObservedRunningTime="2026-07-15 19:45:21.160931051 +0000 UTC m=+435.405437512" Jul 15 19:45:21.229701 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:21.229669 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:21.329949 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:21.329917 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:22.134221 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:22.134188 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8nxrb" Jul 15 19:45:40.881411 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:40.881328 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:40.948989 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:45:40.948955 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:04.375394 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:04.375343 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:04.521572 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:04.521507 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:04.603516 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:04.603465 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:08.605928 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:08.605892 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:08.670523 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:08.670486 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:19.488939 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:19.488902 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:19.589291 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:19.589261 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:19.832763 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:19.832673 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws"] Jul 15 19:46:19.836309 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:19.836291 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws" Jul 15 19:46:19.838878 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:19.838858 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 19:46:19.839031 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:19.838860 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 19:46:19.839031 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:19.838860 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7djq2\"" Jul 15 19:46:19.844371 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:19.843942 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws"] Jul 15 19:46:19.940711 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:19.940672 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9ce92c8-b704-4173-8fd5-20b926a4d18b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws\" (UID: \"e9ce92c8-b704-4173-8fd5-20b926a4d18b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws" Jul 15 19:46:19.940933 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:19.940722 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x99kq\" (UniqueName: \"kubernetes.io/projected/e9ce92c8-b704-4173-8fd5-20b926a4d18b-kube-api-access-x99kq\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws\" (UID: \"e9ce92c8-b704-4173-8fd5-20b926a4d18b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws" Jul 15 19:46:19.940933 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:19.940843 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9ce92c8-b704-4173-8fd5-20b926a4d18b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws\" (UID: \"e9ce92c8-b704-4173-8fd5-20b926a4d18b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws" Jul 15 19:46:20.042151 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.042120 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9ce92c8-b704-4173-8fd5-20b926a4d18b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws\" (UID: \"e9ce92c8-b704-4173-8fd5-20b926a4d18b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws" Jul 15 19:46:20.042310 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.042160 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x99kq\" (UniqueName: \"kubernetes.io/projected/e9ce92c8-b704-4173-8fd5-20b926a4d18b-kube-api-access-x99kq\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws\" (UID: \"e9ce92c8-b704-4173-8fd5-20b926a4d18b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws" Jul 15 19:46:20.042310 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.042204 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9ce92c8-b704-4173-8fd5-20b926a4d18b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws\" (UID: \"e9ce92c8-b704-4173-8fd5-20b926a4d18b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws" Jul 15 19:46:20.042514 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.042495 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9ce92c8-b704-4173-8fd5-20b926a4d18b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws\" (UID: \"e9ce92c8-b704-4173-8fd5-20b926a4d18b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws" Jul 15 19:46:20.042564 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.042542 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9ce92c8-b704-4173-8fd5-20b926a4d18b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws\" (UID: \"e9ce92c8-b704-4173-8fd5-20b926a4d18b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws" Jul 15 19:46:20.051078 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.051053 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x99kq\" (UniqueName: \"kubernetes.io/projected/e9ce92c8-b704-4173-8fd5-20b926a4d18b-kube-api-access-x99kq\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws\" (UID: \"e9ce92c8-b704-4173-8fd5-20b926a4d18b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws" Jul 15 19:46:20.147495 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.147396 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws" Jul 15 19:46:20.287162 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.287129 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws"] Jul 15 19:46:20.290473 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:46:20.290445 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ce92c8_b704_4173_8fd5_20b926a4d18b.slice/crio-fb6e0e20a333e6fba223354ff23adf672ee2bfa67bd6cd99651207f46dde786d WatchSource:0}: Error finding container fb6e0e20a333e6fba223354ff23adf672ee2bfa67bd6cd99651207f46dde786d: Status 404 returned error can't find the container with id fb6e0e20a333e6fba223354ff23adf672ee2bfa67bd6cd99651207f46dde786d Jul 15 19:46:20.408453 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.408415 2554 generic.go:356] "Generic (PLEG): container finished" podID="e9ce92c8-b704-4173-8fd5-20b926a4d18b" containerID="efa86249a2a533e6b41e9abed83fc97edc37db8d4d8f27cd262e17eac04832e9" exitCode=0 Jul 15 19:46:20.408603 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.408467 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws" event={"ID":"e9ce92c8-b704-4173-8fd5-20b926a4d18b","Type":"ContainerDied","Data":"efa86249a2a533e6b41e9abed83fc97edc37db8d4d8f27cd262e17eac04832e9"} Jul 15 19:46:20.408603 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.408511 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws" event={"ID":"e9ce92c8-b704-4173-8fd5-20b926a4d18b","Type":"ContainerStarted","Data":"fb6e0e20a333e6fba223354ff23adf672ee2bfa67bd6cd99651207f46dde786d"} Jul 15 19:46:20.455525 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.455485 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2"] Jul 15 19:46:20.458837 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.458821 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2" Jul 15 19:46:20.468105 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.468077 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2"] Jul 15 19:46:20.548175 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.548134 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfq2f\" (UniqueName: \"kubernetes.io/projected/28945aeb-cc94-41c9-a5ff-9ff37423e55b-kube-api-access-sfq2f\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2\" (UID: \"28945aeb-cc94-41c9-a5ff-9ff37423e55b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2" Jul 15 19:46:20.548575 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.548193 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28945aeb-cc94-41c9-a5ff-9ff37423e55b-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2\" (UID: \"28945aeb-cc94-41c9-a5ff-9ff37423e55b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2" Jul 15 19:46:20.548575 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.548231 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28945aeb-cc94-41c9-a5ff-9ff37423e55b-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2\" (UID: \"28945aeb-cc94-41c9-a5ff-9ff37423e55b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2" Jul 15 19:46:20.649582 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.649473 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfq2f\" (UniqueName: \"kubernetes.io/projected/28945aeb-cc94-41c9-a5ff-9ff37423e55b-kube-api-access-sfq2f\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2\" (UID: \"28945aeb-cc94-41c9-a5ff-9ff37423e55b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2" Jul 15 19:46:20.649582 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.649546 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28945aeb-cc94-41c9-a5ff-9ff37423e55b-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2\" (UID: \"28945aeb-cc94-41c9-a5ff-9ff37423e55b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2" Jul 15 19:46:20.649875 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.649704 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28945aeb-cc94-41c9-a5ff-9ff37423e55b-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2\" (UID: \"28945aeb-cc94-41c9-a5ff-9ff37423e55b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2" Jul 15 19:46:20.650012 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.649990 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28945aeb-cc94-41c9-a5ff-9ff37423e55b-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2\" (UID: \"28945aeb-cc94-41c9-a5ff-9ff37423e55b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2" Jul 15 19:46:20.650077 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.650020 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28945aeb-cc94-41c9-a5ff-9ff37423e55b-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2\" (UID: \"28945aeb-cc94-41c9-a5ff-9ff37423e55b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2" Jul 15 19:46:20.658668 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.658642 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfq2f\" (UniqueName: \"kubernetes.io/projected/28945aeb-cc94-41c9-a5ff-9ff37423e55b-kube-api-access-sfq2f\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2\" (UID: \"28945aeb-cc94-41c9-a5ff-9ff37423e55b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2" Jul 15 19:46:20.768837 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.768802 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2" Jul 15 19:46:20.921011 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:20.920988 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2"] Jul 15 19:46:20.922837 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:46:20.922809 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28945aeb_cc94_41c9_a5ff_9ff37423e55b.slice/crio-45fe23d08600538470dce8923ffc20812b15af2c96f6fbad8627eb6911946915 WatchSource:0}: Error finding container 45fe23d08600538470dce8923ffc20812b15af2c96f6fbad8627eb6911946915: Status 404 returned error can't find the container with id 45fe23d08600538470dce8923ffc20812b15af2c96f6fbad8627eb6911946915 Jul 15 19:46:21.235547 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.235522 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr"] Jul 15 19:46:21.240251 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.240232 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr" Jul 15 19:46:21.253226 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.253196 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr"] Jul 15 19:46:21.255129 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.255106 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d0a135d-3c27-4f90-9b18-1d40028588cd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr\" (UID: \"5d0a135d-3c27-4f90-9b18-1d40028588cd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr" Jul 15 19:46:21.255265 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.255176 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6lx9\" (UniqueName: \"kubernetes.io/projected/5d0a135d-3c27-4f90-9b18-1d40028588cd-kube-api-access-j6lx9\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr\" (UID: \"5d0a135d-3c27-4f90-9b18-1d40028588cd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr" Jul 15 19:46:21.255265 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.255220 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d0a135d-3c27-4f90-9b18-1d40028588cd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr\" (UID: \"5d0a135d-3c27-4f90-9b18-1d40028588cd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr" Jul 15 19:46:21.356310 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.356276 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6lx9\" (UniqueName: \"kubernetes.io/projected/5d0a135d-3c27-4f90-9b18-1d40028588cd-kube-api-access-j6lx9\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr\" (UID: \"5d0a135d-3c27-4f90-9b18-1d40028588cd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr" Jul 15 19:46:21.356310 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.356319 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d0a135d-3c27-4f90-9b18-1d40028588cd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr\" (UID: \"5d0a135d-3c27-4f90-9b18-1d40028588cd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr" Jul 15 19:46:21.356511 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.356391 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d0a135d-3c27-4f90-9b18-1d40028588cd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr\" (UID: \"5d0a135d-3c27-4f90-9b18-1d40028588cd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr" Jul 15 19:46:21.356821 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.356799 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d0a135d-3c27-4f90-9b18-1d40028588cd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr\" (UID: \"5d0a135d-3c27-4f90-9b18-1d40028588cd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr" Jul 15 19:46:21.356899 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.356835 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d0a135d-3c27-4f90-9b18-1d40028588cd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr\" (UID: \"5d0a135d-3c27-4f90-9b18-1d40028588cd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr" Jul 15 19:46:21.365180 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.365158 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6lx9\" (UniqueName: \"kubernetes.io/projected/5d0a135d-3c27-4f90-9b18-1d40028588cd-kube-api-access-j6lx9\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr\" (UID: \"5d0a135d-3c27-4f90-9b18-1d40028588cd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr" Jul 15 19:46:21.415658 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.415624 2554 generic.go:356] "Generic (PLEG): container finished" podID="28945aeb-cc94-41c9-a5ff-9ff37423e55b" containerID="3c8d882ec2dd70f89d9b5bd5d0e8c70e418c941b399e358624e83ca3acbd801a" exitCode=0 Jul 15 19:46:21.415846 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.415715 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2" event={"ID":"28945aeb-cc94-41c9-a5ff-9ff37423e55b","Type":"ContainerDied","Data":"3c8d882ec2dd70f89d9b5bd5d0e8c70e418c941b399e358624e83ca3acbd801a"} Jul 15 19:46:21.415846 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.415789 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2" event={"ID":"28945aeb-cc94-41c9-a5ff-9ff37423e55b","Type":"ContainerStarted","Data":"45fe23d08600538470dce8923ffc20812b15af2c96f6fbad8627eb6911946915"} Jul 15 19:46:21.417730 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.417705 2554 generic.go:356] "Generic (PLEG): container finished" podID="e9ce92c8-b704-4173-8fd5-20b926a4d18b" containerID="fa6e55e5a9fc8eb55a8894b0cdba1254e6278ee7ec2bf706c702b09dd98d9d4b" exitCode=0 Jul 15 19:46:21.417819 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.417778 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws" event={"ID":"e9ce92c8-b704-4173-8fd5-20b926a4d18b","Type":"ContainerDied","Data":"fa6e55e5a9fc8eb55a8894b0cdba1254e6278ee7ec2bf706c702b09dd98d9d4b"} Jul 15 19:46:21.562601 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.562558 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr" Jul 15 19:46:21.699373 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.699346 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr"] Jul 15 19:46:21.701455 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:46:21.701424 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0a135d_3c27_4f90_9b18_1d40028588cd.slice/crio-1d4d74eddd6f89a118ea67bdb9798d0f790eb080209950071ff308121ab93888 WatchSource:0}: Error finding container 1d4d74eddd6f89a118ea67bdb9798d0f790eb080209950071ff308121ab93888: Status 404 returned error can't find the container with id 1d4d74eddd6f89a118ea67bdb9798d0f790eb080209950071ff308121ab93888 Jul 15 19:46:21.838192 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.838155 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs"] Jul 15 19:46:21.841597 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.841578 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs" Jul 15 19:46:21.851499 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.851470 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs"] Jul 15 19:46:21.861369 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.861337 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a77b95a-586b-42ee-bb28-7125e1df08ca-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs\" (UID: \"4a77b95a-586b-42ee-bb28-7125e1df08ca\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs" Jul 15 19:46:21.861513 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.861393 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a77b95a-586b-42ee-bb28-7125e1df08ca-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs\" (UID: \"4a77b95a-586b-42ee-bb28-7125e1df08ca\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs" Jul 15 19:46:21.861513 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.861470 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8brwr\" (UniqueName: \"kubernetes.io/projected/4a77b95a-586b-42ee-bb28-7125e1df08ca-kube-api-access-8brwr\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs\" (UID: \"4a77b95a-586b-42ee-bb28-7125e1df08ca\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs" Jul 15 19:46:21.962479 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.962389 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a77b95a-586b-42ee-bb28-7125e1df08ca-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs\" (UID: \"4a77b95a-586b-42ee-bb28-7125e1df08ca\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs" Jul 15 19:46:21.962479 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.962431 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a77b95a-586b-42ee-bb28-7125e1df08ca-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs\" (UID: \"4a77b95a-586b-42ee-bb28-7125e1df08ca\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs" Jul 15 19:46:21.962479 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.962455 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8brwr\" (UniqueName: \"kubernetes.io/projected/4a77b95a-586b-42ee-bb28-7125e1df08ca-kube-api-access-8brwr\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs\" (UID: \"4a77b95a-586b-42ee-bb28-7125e1df08ca\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs" Jul 15 19:46:21.962923 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.962899 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a77b95a-586b-42ee-bb28-7125e1df08ca-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs\" (UID: \"4a77b95a-586b-42ee-bb28-7125e1df08ca\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs" Jul 15 19:46:21.962967 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.962910 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a77b95a-586b-42ee-bb28-7125e1df08ca-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs\" (UID: \"4a77b95a-586b-42ee-bb28-7125e1df08ca\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs" Jul 15 19:46:21.971246 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:21.971223 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8brwr\" (UniqueName: \"kubernetes.io/projected/4a77b95a-586b-42ee-bb28-7125e1df08ca-kube-api-access-8brwr\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs\" (UID: \"4a77b95a-586b-42ee-bb28-7125e1df08ca\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs" Jul 15 19:46:22.151386 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:22.151357 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs" Jul 15 19:46:22.302467 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:22.301873 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs"] Jul 15 19:46:22.306210 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:46:22.306175 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a77b95a_586b_42ee_bb28_7125e1df08ca.slice/crio-f298eba377776ab8342ad86cae79dd91648c855d78cfc51e00cd8546636185a3 WatchSource:0}: Error finding container f298eba377776ab8342ad86cae79dd91648c855d78cfc51e00cd8546636185a3: Status 404 returned error can't find the container with id f298eba377776ab8342ad86cae79dd91648c855d78cfc51e00cd8546636185a3 Jul 15 19:46:22.424777 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:22.424731 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs" event={"ID":"4a77b95a-586b-42ee-bb28-7125e1df08ca","Type":"ContainerStarted","Data":"452e48ca365d6a4af1c306a32b5a05ac0c912fb8cba0f57162290333d0d04bd6"} Jul 15 19:46:22.424976 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:22.424805 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs" event={"ID":"4a77b95a-586b-42ee-bb28-7125e1df08ca","Type":"ContainerStarted","Data":"f298eba377776ab8342ad86cae79dd91648c855d78cfc51e00cd8546636185a3"} Jul 15 19:46:22.426659 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:22.426637 2554 generic.go:356] "Generic (PLEG): container finished" podID="5d0a135d-3c27-4f90-9b18-1d40028588cd" containerID="ea34389c43035d7d1cc6ab9f362c3e6987d594772c06ab8a7a66d58b64ba72a1" exitCode=0 Jul 15 19:46:22.426815 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:22.426717 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr" event={"ID":"5d0a135d-3c27-4f90-9b18-1d40028588cd","Type":"ContainerDied","Data":"ea34389c43035d7d1cc6ab9f362c3e6987d594772c06ab8a7a66d58b64ba72a1"} Jul 15 19:46:22.426815 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:22.426790 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr" event={"ID":"5d0a135d-3c27-4f90-9b18-1d40028588cd","Type":"ContainerStarted","Data":"1d4d74eddd6f89a118ea67bdb9798d0f790eb080209950071ff308121ab93888"} Jul 15 19:46:22.428907 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:22.428878 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2" event={"ID":"28945aeb-cc94-41c9-a5ff-9ff37423e55b","Type":"ContainerStarted","Data":"591e3faa867d0774718cd7c0a0373fdc2dc2a388421ff77cb6dcc0589044d991"} Jul 15 19:46:22.431691 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:22.431667 2554 generic.go:356] "Generic (PLEG): container finished" podID="e9ce92c8-b704-4173-8fd5-20b926a4d18b" containerID="1c32e1ad4f64572dbd758d51d03c0651a891f54a63646745f9ff0ef9299c7dd4" exitCode=0 Jul 15 19:46:22.431817 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:22.431722 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws" event={"ID":"e9ce92c8-b704-4173-8fd5-20b926a4d18b","Type":"ContainerDied","Data":"1c32e1ad4f64572dbd758d51d03c0651a891f54a63646745f9ff0ef9299c7dd4"} Jul 15 19:46:23.439575 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:23.439536 2554 generic.go:356] "Generic (PLEG): container finished" podID="4a77b95a-586b-42ee-bb28-7125e1df08ca" containerID="452e48ca365d6a4af1c306a32b5a05ac0c912fb8cba0f57162290333d0d04bd6" exitCode=0 Jul 15 19:46:23.440058 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:23.439576 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs" event={"ID":"4a77b95a-586b-42ee-bb28-7125e1df08ca","Type":"ContainerDied","Data":"452e48ca365d6a4af1c306a32b5a05ac0c912fb8cba0f57162290333d0d04bd6"} Jul 15 19:46:23.441686 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:23.441662 2554 generic.go:356] "Generic (PLEG): container finished" podID="5d0a135d-3c27-4f90-9b18-1d40028588cd" containerID="f39dccddc7c9a041e301137fcebad0b5626ba77f265bdd57db94d2a70f869f45" exitCode=0 Jul 15 19:46:23.441837 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:23.441734 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr" event={"ID":"5d0a135d-3c27-4f90-9b18-1d40028588cd","Type":"ContainerDied","Data":"f39dccddc7c9a041e301137fcebad0b5626ba77f265bdd57db94d2a70f869f45"} Jul 15 19:46:23.443696 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:23.443675 2554 generic.go:356] "Generic (PLEG): container finished" podID="28945aeb-cc94-41c9-a5ff-9ff37423e55b" containerID="591e3faa867d0774718cd7c0a0373fdc2dc2a388421ff77cb6dcc0589044d991" exitCode=0 Jul 15 19:46:23.443808 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:23.443775 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2" event={"ID":"28945aeb-cc94-41c9-a5ff-9ff37423e55b","Type":"ContainerDied","Data":"591e3faa867d0774718cd7c0a0373fdc2dc2a388421ff77cb6dcc0589044d991"} Jul 15 19:46:23.570267 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:23.570241 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws" Jul 15 19:46:23.678823 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:23.678791 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9ce92c8-b704-4173-8fd5-20b926a4d18b-bundle\") pod \"e9ce92c8-b704-4173-8fd5-20b926a4d18b\" (UID: \"e9ce92c8-b704-4173-8fd5-20b926a4d18b\") " Jul 15 19:46:23.678954 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:23.678838 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x99kq\" (UniqueName: \"kubernetes.io/projected/e9ce92c8-b704-4173-8fd5-20b926a4d18b-kube-api-access-x99kq\") pod \"e9ce92c8-b704-4173-8fd5-20b926a4d18b\" (UID: \"e9ce92c8-b704-4173-8fd5-20b926a4d18b\") " Jul 15 19:46:23.678954 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:23.678893 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9ce92c8-b704-4173-8fd5-20b926a4d18b-util\") pod \"e9ce92c8-b704-4173-8fd5-20b926a4d18b\" (UID: \"e9ce92c8-b704-4173-8fd5-20b926a4d18b\") " Jul 15 19:46:23.679432 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:23.679407 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ce92c8-b704-4173-8fd5-20b926a4d18b-bundle" (OuterVolumeSpecName: "bundle") pod "e9ce92c8-b704-4173-8fd5-20b926a4d18b" (UID: "e9ce92c8-b704-4173-8fd5-20b926a4d18b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:46:23.681261 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:23.681239 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ce92c8-b704-4173-8fd5-20b926a4d18b-kube-api-access-x99kq" (OuterVolumeSpecName: "kube-api-access-x99kq") pod "e9ce92c8-b704-4173-8fd5-20b926a4d18b" (UID: "e9ce92c8-b704-4173-8fd5-20b926a4d18b"). InnerVolumeSpecName "kube-api-access-x99kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:46:23.684186 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:23.684154 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ce92c8-b704-4173-8fd5-20b926a4d18b-util" (OuterVolumeSpecName: "util") pod "e9ce92c8-b704-4173-8fd5-20b926a4d18b" (UID: "e9ce92c8-b704-4173-8fd5-20b926a4d18b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:46:23.779766 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:23.779717 2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9ce92c8-b704-4173-8fd5-20b926a4d18b-util\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:46:23.779766 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:23.779746 2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9ce92c8-b704-4173-8fd5-20b926a4d18b-bundle\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:46:23.779953 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:23.779778 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x99kq\" (UniqueName: \"kubernetes.io/projected/e9ce92c8-b704-4173-8fd5-20b926a4d18b-kube-api-access-x99kq\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:46:24.450851 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:24.450824 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws" Jul 15 19:46:24.451240 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:24.450848 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws" event={"ID":"e9ce92c8-b704-4173-8fd5-20b926a4d18b","Type":"ContainerDied","Data":"fb6e0e20a333e6fba223354ff23adf672ee2bfa67bd6cd99651207f46dde786d"} Jul 15 19:46:24.451240 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:24.450895 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb6e0e20a333e6fba223354ff23adf672ee2bfa67bd6cd99651207f46dde786d" Jul 15 19:46:24.452780 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:24.452734 2554 generic.go:356] "Generic (PLEG): container finished" podID="4a77b95a-586b-42ee-bb28-7125e1df08ca" containerID="09aa98f24346ff3fc07f857403dd6e43438fa180f83178d8374614f1ea479e56" exitCode=0 Jul 15 19:46:24.452906 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:24.452797 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs" event={"ID":"4a77b95a-586b-42ee-bb28-7125e1df08ca","Type":"ContainerDied","Data":"09aa98f24346ff3fc07f857403dd6e43438fa180f83178d8374614f1ea479e56"} Jul 15 19:46:24.455222 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:24.455200 2554 generic.go:356] "Generic (PLEG): container finished" podID="5d0a135d-3c27-4f90-9b18-1d40028588cd" containerID="e2f03b8de680df12e7e0d31804cc1341bf6e0b896c3534f54a47ada8a0ee2675" exitCode=0 Jul 15 19:46:24.455318 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:24.455232 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr" event={"ID":"5d0a135d-3c27-4f90-9b18-1d40028588cd","Type":"ContainerDied","Data":"e2f03b8de680df12e7e0d31804cc1341bf6e0b896c3534f54a47ada8a0ee2675"} Jul 15 19:46:24.457343 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:24.457322 2554 generic.go:356] "Generic (PLEG): container finished" podID="28945aeb-cc94-41c9-a5ff-9ff37423e55b" containerID="8fa44f08d49c4f074298b336c90ed8fee2280a6814422bd4b24b79a235fa84d0" exitCode=0 Jul 15 19:46:24.457433 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:24.457349 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2" event={"ID":"28945aeb-cc94-41c9-a5ff-9ff37423e55b","Type":"ContainerDied","Data":"8fa44f08d49c4f074298b336c90ed8fee2280a6814422bd4b24b79a235fa84d0"} Jul 15 19:46:25.464228 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.464192 2554 generic.go:356] "Generic (PLEG): container finished" podID="4a77b95a-586b-42ee-bb28-7125e1df08ca" containerID="d2ed367ae8611dd7b4ee79eb63071f481048721b6e5dd7e597d7815fea140540" exitCode=0 Jul 15 19:46:25.464657 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.464290 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs" event={"ID":"4a77b95a-586b-42ee-bb28-7125e1df08ca","Type":"ContainerDied","Data":"d2ed367ae8611dd7b4ee79eb63071f481048721b6e5dd7e597d7815fea140540"} Jul 15 19:46:25.623268 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.623243 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2" Jul 15 19:46:25.625885 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.625865 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr" Jul 15 19:46:25.697347 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.697317 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d0a135d-3c27-4f90-9b18-1d40028588cd-util\") pod \"5d0a135d-3c27-4f90-9b18-1d40028588cd\" (UID: \"5d0a135d-3c27-4f90-9b18-1d40028588cd\") " Jul 15 19:46:25.697521 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.697363 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfq2f\" (UniqueName: \"kubernetes.io/projected/28945aeb-cc94-41c9-a5ff-9ff37423e55b-kube-api-access-sfq2f\") pod \"28945aeb-cc94-41c9-a5ff-9ff37423e55b\" (UID: \"28945aeb-cc94-41c9-a5ff-9ff37423e55b\") " Jul 15 19:46:25.697521 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.697419 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28945aeb-cc94-41c9-a5ff-9ff37423e55b-util\") pod \"28945aeb-cc94-41c9-a5ff-9ff37423e55b\" (UID: \"28945aeb-cc94-41c9-a5ff-9ff37423e55b\") " Jul 15 19:46:25.697521 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.697450 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6lx9\" (UniqueName: \"kubernetes.io/projected/5d0a135d-3c27-4f90-9b18-1d40028588cd-kube-api-access-j6lx9\") pod \"5d0a135d-3c27-4f90-9b18-1d40028588cd\" (UID: \"5d0a135d-3c27-4f90-9b18-1d40028588cd\") " Jul 15 19:46:25.697521 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.697493 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d0a135d-3c27-4f90-9b18-1d40028588cd-bundle\") pod \"5d0a135d-3c27-4f90-9b18-1d40028588cd\" (UID: \"5d0a135d-3c27-4f90-9b18-1d40028588cd\") " Jul 15 19:46:25.697744 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.697523 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28945aeb-cc94-41c9-a5ff-9ff37423e55b-bundle\") pod \"28945aeb-cc94-41c9-a5ff-9ff37423e55b\" (UID: \"28945aeb-cc94-41c9-a5ff-9ff37423e55b\") " Jul 15 19:46:25.698102 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.698055 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0a135d-3c27-4f90-9b18-1d40028588cd-bundle" (OuterVolumeSpecName: "bundle") pod "5d0a135d-3c27-4f90-9b18-1d40028588cd" (UID: "5d0a135d-3c27-4f90-9b18-1d40028588cd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:46:25.698406 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.698133 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28945aeb-cc94-41c9-a5ff-9ff37423e55b-bundle" (OuterVolumeSpecName: "bundle") pod "28945aeb-cc94-41c9-a5ff-9ff37423e55b" (UID: "28945aeb-cc94-41c9-a5ff-9ff37423e55b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:46:25.699866 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.699834 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0a135d-3c27-4f90-9b18-1d40028588cd-kube-api-access-j6lx9" (OuterVolumeSpecName: "kube-api-access-j6lx9") pod "5d0a135d-3c27-4f90-9b18-1d40028588cd" (UID: "5d0a135d-3c27-4f90-9b18-1d40028588cd"). InnerVolumeSpecName "kube-api-access-j6lx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:46:25.700110 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.700091 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28945aeb-cc94-41c9-a5ff-9ff37423e55b-kube-api-access-sfq2f" (OuterVolumeSpecName: "kube-api-access-sfq2f") pod "28945aeb-cc94-41c9-a5ff-9ff37423e55b" (UID: "28945aeb-cc94-41c9-a5ff-9ff37423e55b"). InnerVolumeSpecName "kube-api-access-sfq2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:46:25.703435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.703411 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28945aeb-cc94-41c9-a5ff-9ff37423e55b-util" (OuterVolumeSpecName: "util") pod "28945aeb-cc94-41c9-a5ff-9ff37423e55b" (UID: "28945aeb-cc94-41c9-a5ff-9ff37423e55b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:46:25.704603 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.704582 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0a135d-3c27-4f90-9b18-1d40028588cd-util" (OuterVolumeSpecName: "util") pod "5d0a135d-3c27-4f90-9b18-1d40028588cd" (UID: "5d0a135d-3c27-4f90-9b18-1d40028588cd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:46:25.798718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.798632 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfq2f\" (UniqueName: \"kubernetes.io/projected/28945aeb-cc94-41c9-a5ff-9ff37423e55b-kube-api-access-sfq2f\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:46:25.798718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.798663 2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28945aeb-cc94-41c9-a5ff-9ff37423e55b-util\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:46:25.798718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.798675 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6lx9\" (UniqueName: \"kubernetes.io/projected/5d0a135d-3c27-4f90-9b18-1d40028588cd-kube-api-access-j6lx9\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:46:25.798718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.798684 2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d0a135d-3c27-4f90-9b18-1d40028588cd-bundle\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:46:25.798718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.798693 2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28945aeb-cc94-41c9-a5ff-9ff37423e55b-bundle\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:46:25.798718 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:25.798701 2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d0a135d-3c27-4f90-9b18-1d40028588cd-util\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:46:26.471819 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:26.471785 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr" event={"ID":"5d0a135d-3c27-4f90-9b18-1d40028588cd","Type":"ContainerDied","Data":"1d4d74eddd6f89a118ea67bdb9798d0f790eb080209950071ff308121ab93888"} Jul 15 19:46:26.472273 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:26.471849 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d4d74eddd6f89a118ea67bdb9798d0f790eb080209950071ff308121ab93888" Jul 15 19:46:26.472273 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:26.471796 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr" Jul 15 19:46:26.473941 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:26.473823 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2" event={"ID":"28945aeb-cc94-41c9-a5ff-9ff37423e55b","Type":"ContainerDied","Data":"45fe23d08600538470dce8923ffc20812b15af2c96f6fbad8627eb6911946915"} Jul 15 19:46:26.473941 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:26.473871 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45fe23d08600538470dce8923ffc20812b15af2c96f6fbad8627eb6911946915" Jul 15 19:46:26.474138 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:26.474061 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2" Jul 15 19:46:26.595617 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:26.595591 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs" Jul 15 19:46:26.706058 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:26.706023 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a77b95a-586b-42ee-bb28-7125e1df08ca-bundle\") pod \"4a77b95a-586b-42ee-bb28-7125e1df08ca\" (UID: \"4a77b95a-586b-42ee-bb28-7125e1df08ca\") " Jul 15 19:46:26.706250 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:26.706075 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a77b95a-586b-42ee-bb28-7125e1df08ca-util\") pod \"4a77b95a-586b-42ee-bb28-7125e1df08ca\" (UID: \"4a77b95a-586b-42ee-bb28-7125e1df08ca\") " Jul 15 19:46:26.706250 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:26.706110 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8brwr\" (UniqueName: \"kubernetes.io/projected/4a77b95a-586b-42ee-bb28-7125e1df08ca-kube-api-access-8brwr\") pod \"4a77b95a-586b-42ee-bb28-7125e1df08ca\" (UID: \"4a77b95a-586b-42ee-bb28-7125e1df08ca\") " Jul 15 19:46:26.706507 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:26.706484 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a77b95a-586b-42ee-bb28-7125e1df08ca-bundle" (OuterVolumeSpecName: "bundle") pod "4a77b95a-586b-42ee-bb28-7125e1df08ca" (UID: "4a77b95a-586b-42ee-bb28-7125e1df08ca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:46:26.708535 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:26.708513 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a77b95a-586b-42ee-bb28-7125e1df08ca-kube-api-access-8brwr" (OuterVolumeSpecName: "kube-api-access-8brwr") pod "4a77b95a-586b-42ee-bb28-7125e1df08ca" (UID: "4a77b95a-586b-42ee-bb28-7125e1df08ca"). InnerVolumeSpecName "kube-api-access-8brwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:46:26.711503 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:26.711481 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a77b95a-586b-42ee-bb28-7125e1df08ca-util" (OuterVolumeSpecName: "util") pod "4a77b95a-586b-42ee-bb28-7125e1df08ca" (UID: "4a77b95a-586b-42ee-bb28-7125e1df08ca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:46:26.807670 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:26.807568 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8brwr\" (UniqueName: \"kubernetes.io/projected/4a77b95a-586b-42ee-bb28-7125e1df08ca-kube-api-access-8brwr\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:46:26.807670 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:26.807613 2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a77b95a-586b-42ee-bb28-7125e1df08ca-bundle\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:46:26.807670 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:26.807623 2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a77b95a-586b-42ee-bb28-7125e1df08ca-util\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 19:46:27.480319 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:27.480290 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs" Jul 15 19:46:27.480684 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:27.480320 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs" event={"ID":"4a77b95a-586b-42ee-bb28-7125e1df08ca","Type":"ContainerDied","Data":"f298eba377776ab8342ad86cae79dd91648c855d78cfc51e00cd8546636185a3"} Jul 15 19:46:27.480684 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:27.480364 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f298eba377776ab8342ad86cae79dd91648c855d78cfc51e00cd8546636185a3" Jul 15 19:46:40.083966 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:40.083927 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:43.361358 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:43.361273 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:47.139272 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:47.139238 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:48.285037 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:48.285006 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:48.627357 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:48.627260 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:48.881783 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:48.881680 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:51.953040 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:51.953004 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:52.153524 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:52.153492 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:54.722196 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:54.722158 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:56.090839 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:56.090804 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:56.192264 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:56.192228 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:58.206000 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:58.205915 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:58.307571 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:58.307527 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:59.683492 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:59.683454 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:59.783226 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:46:59.783181 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:08.061166 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:08.061125 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:08.195354 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:08.195318 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:09.175777 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:09.175716 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:09.275519 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:09.275489 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:12.158809 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:12.158779 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:12.268427 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:12.268392 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:14.622590 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:14.622549 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:14.723139 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:14.723105 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:18.055151 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:18.055111 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:18.155229 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:18.155197 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:26.930703 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:26.930666 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:27.070994 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:27.070958 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:29.896985 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:29.896955 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:29.999697 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:29.999657 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:32.520240 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:32.520208 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:32.627159 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:32.627115 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:36.296618 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:36.296575 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:36.362064 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:36.362031 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:42.633384 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:42.633347 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:42.919517 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:42.919485 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:43.699049 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:43.699009 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:43.804767 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:43.804711 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:44.849400 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:44.849362 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:44.955238 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:44.955199 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:45.071167 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.071132 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:45.241284 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241248 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:47:45.241613 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241596 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28945aeb-cc94-41c9-a5ff-9ff37423e55b" containerName="extract" Jul 15 19:47:45.241694 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241615 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="28945aeb-cc94-41c9-a5ff-9ff37423e55b" containerName="extract" Jul 15 19:47:45.241694 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241633 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d0a135d-3c27-4f90-9b18-1d40028588cd" containerName="util" Jul 15 19:47:45.241694 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241642 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d0a135d-3c27-4f90-9b18-1d40028588cd" containerName="util" Jul 15 19:47:45.241694 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241652 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28945aeb-cc94-41c9-a5ff-9ff37423e55b" containerName="pull" Jul 15 19:47:45.241694 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241660 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="28945aeb-cc94-41c9-a5ff-9ff37423e55b" containerName="pull" Jul 15 19:47:45.241694 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241672 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d0a135d-3c27-4f90-9b18-1d40028588cd" containerName="pull" Jul 15 19:47:45.241694 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241680 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d0a135d-3c27-4f90-9b18-1d40028588cd" containerName="pull" Jul 15 19:47:45.241694 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241696 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a77b95a-586b-42ee-bb28-7125e1df08ca" containerName="extract" Jul 15 19:47:45.242100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241704 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a77b95a-586b-42ee-bb28-7125e1df08ca" containerName="extract" Jul 15 19:47:45.242100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241713 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ce92c8-b704-4173-8fd5-20b926a4d18b" containerName="util" Jul 15 19:47:45.242100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241721 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ce92c8-b704-4173-8fd5-20b926a4d18b" containerName="util" Jul 15 19:47:45.242100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241731 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ce92c8-b704-4173-8fd5-20b926a4d18b" containerName="extract" Jul 15 19:47:45.242100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241741 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ce92c8-b704-4173-8fd5-20b926a4d18b" containerName="extract" Jul 15 19:47:45.242100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241778 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a77b95a-586b-42ee-bb28-7125e1df08ca" containerName="util" Jul 15 19:47:45.242100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241787 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a77b95a-586b-42ee-bb28-7125e1df08ca" containerName="util" Jul 15 19:47:45.242100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241803 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28945aeb-cc94-41c9-a5ff-9ff37423e55b" containerName="util" Jul 15 19:47:45.242100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241812 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="28945aeb-cc94-41c9-a5ff-9ff37423e55b" containerName="util" Jul 15 19:47:45.242100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241823 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d0a135d-3c27-4f90-9b18-1d40028588cd" containerName="extract" Jul 15 19:47:45.242100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241831 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d0a135d-3c27-4f90-9b18-1d40028588cd" containerName="extract" Jul 15 19:47:45.242100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241842 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ce92c8-b704-4173-8fd5-20b926a4d18b" containerName="pull" Jul 15 19:47:45.242100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241850 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ce92c8-b704-4173-8fd5-20b926a4d18b" containerName="pull" Jul 15 19:47:45.242100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241858 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a77b95a-586b-42ee-bb28-7125e1df08ca" containerName="pull" Jul 15 19:47:45.242100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241866 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a77b95a-586b-42ee-bb28-7125e1df08ca" containerName="pull" Jul 15 19:47:45.242100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241958 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ce92c8-b704-4173-8fd5-20b926a4d18b" containerName="extract" Jul 15 19:47:45.242100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241969 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d0a135d-3c27-4f90-9b18-1d40028588cd" containerName="extract" Jul 15 19:47:45.242100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241983 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a77b95a-586b-42ee-bb28-7125e1df08ca" containerName="extract" Jul 15 19:47:45.242100 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.241992 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="28945aeb-cc94-41c9-a5ff-9ff37423e55b" containerName="extract" Jul 15 19:47:45.245209 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.245187 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ggt9w" Jul 15 19:47:45.247762 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.247731 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 15 19:47:45.247864 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.247778 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 19:47:45.247864 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.247777 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 19:47:45.247864 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.247731 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7djq2\"" Jul 15 19:47:45.254624 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.254601 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:47:45.295302 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.295268 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:47:45.366817 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.366746 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e5b93a46-4b97-48ea-861c-e6ca6a9ffdef-config-file\") pod \"limitador-limitador-54d6d988d5-ggt9w\" (UID: \"e5b93a46-4b97-48ea-861c-e6ca6a9ffdef\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ggt9w" Jul 15 19:47:45.367014 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.366874 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmn2x\" (UniqueName: \"kubernetes.io/projected/e5b93a46-4b97-48ea-861c-e6ca6a9ffdef-kube-api-access-pmn2x\") pod \"limitador-limitador-54d6d988d5-ggt9w\" (UID: \"e5b93a46-4b97-48ea-861c-e6ca6a9ffdef\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ggt9w" Jul 15 19:47:45.467435 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.467395 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmn2x\" (UniqueName: \"kubernetes.io/projected/e5b93a46-4b97-48ea-861c-e6ca6a9ffdef-kube-api-access-pmn2x\") pod \"limitador-limitador-54d6d988d5-ggt9w\" (UID: \"e5b93a46-4b97-48ea-861c-e6ca6a9ffdef\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ggt9w" Jul 15 19:47:45.467639 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.467455 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e5b93a46-4b97-48ea-861c-e6ca6a9ffdef-config-file\") pod \"limitador-limitador-54d6d988d5-ggt9w\" (UID: \"e5b93a46-4b97-48ea-861c-e6ca6a9ffdef\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ggt9w" Jul 15 19:47:45.468055 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.468036 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e5b93a46-4b97-48ea-861c-e6ca6a9ffdef-config-file\") pod \"limitador-limitador-54d6d988d5-ggt9w\" (UID: \"e5b93a46-4b97-48ea-861c-e6ca6a9ffdef\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ggt9w" Jul 15 19:47:45.476252 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.476230 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmn2x\" (UniqueName: \"kubernetes.io/projected/e5b93a46-4b97-48ea-861c-e6ca6a9ffdef-kube-api-access-pmn2x\") pod \"limitador-limitador-54d6d988d5-ggt9w\" (UID: \"e5b93a46-4b97-48ea-861c-e6ca6a9ffdef\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ggt9w" Jul 15 19:47:45.556411 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.556325 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ggt9w" Jul 15 19:47:45.696027 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.696004 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:47:45.697783 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:47:45.697738 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b93a46_4b97_48ea_861c_e6ca6a9ffdef.slice/crio-4f5f812f6dfabae35c353260172a08151c9bfce3121a845b83df0f5761e169bf WatchSource:0}: Error finding container 4f5f812f6dfabae35c353260172a08151c9bfce3121a845b83df0f5761e169bf: Status 404 returned error can't find the container with id 4f5f812f6dfabae35c353260172a08151c9bfce3121a845b83df0f5761e169bf Jul 15 19:47:45.701689 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.701662 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:45.764203 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.764169 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:45.868867 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:45.868746 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ggt9w" event={"ID":"e5b93a46-4b97-48ea-861c-e6ca6a9ffdef","Type":"ContainerStarted","Data":"4f5f812f6dfabae35c353260172a08151c9bfce3121a845b83df0f5761e169bf"} Jul 15 19:47:46.142622 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:46.142546 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:46.298884 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:46.298822 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:48.887463 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:48.887425 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ggt9w" event={"ID":"e5b93a46-4b97-48ea-861c-e6ca6a9ffdef","Type":"ContainerStarted","Data":"c2d1e0c3af329074e87e6eb46e6a25602d62703e7258d8f8ff43caa094a0f2b1"} Jul 15 19:47:48.887876 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:48.887492 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ggt9w" Jul 15 19:47:48.906120 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:48.906066 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-ggt9w" podStartSLOduration=1.424704382 podStartE2EDuration="3.906049477s" totalImagesPullingTime="2.481345095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:47:48.904382466 +0000 UTC m=+583.148888928" watchObservedRunningTime="2026-07-15 19:47:48.906049477 +0000 UTC m=+583.150555939" Jul 15 19:47:49.506734 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:49.506703 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:49.609135 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:49.609099 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:58.743195 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:58.743155 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:59.894500 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:47:59.894469 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ggt9w" Jul 15 19:48:01.075629 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:48:01.075595 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:48:01.136416 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:48:01.136388 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:48:06.045369 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:48:06.045325 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:48:06.140205 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:48:06.140164 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:48:06.209345 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:48:06.209311 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 19:48:06.209527 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:48:06.209425 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 19:48:06.213636 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:48:06.213617 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 19:48:06.213838 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:48:06.213819 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 19:48:06.536101 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:48:06.536066 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:48:21.356453 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:48:21.356414 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:48:21.606497 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:48:21.606415 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:48:23.314853 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:48:23.314815 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:48:28.378089 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:48:28.377992 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:48:30.513526 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:48:30.513471 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:48:32.338712 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:48:32.338669 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:48:32.437364 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:48:32.437333 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:49:05.192863 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:05.192809 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:49:05.292797 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:05.292744 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:49:05.760455 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:05.760421 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:49:06.040733 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:06.040640 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:49:06.417335 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:06.417232 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:49:06.529496 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:06.529406 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:49:06.705820 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:06.705788 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:49:12.096426 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:12.096389 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:49:12.213142 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:12.213100 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:49:15.915425 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:15.915390 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:49:16.095097 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:16.095056 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:49:39.279158 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:39.279114 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:49:39.677332 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:39.677301 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:49:55.329324 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.329286 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv"] Jul 15 19:49:55.332926 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.332901 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.335802 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.335780 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 19:49:55.335912 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.335880 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 19:49:55.336955 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.336937 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nmvtf\"" Jul 15 19:49:55.337067 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.336984 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 15 19:49:55.346693 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.346668 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv"] Jul 15 19:49:55.358595 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.358558 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:49:55.389524 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.389487 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f95134-bc21-41b6-870d-980b333968ba-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxqkv\" (UID: \"c6f95134-bc21-41b6-870d-980b333968ba\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.389708 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.389537 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccx2s\" (UniqueName: \"kubernetes.io/projected/c6f95134-bc21-41b6-870d-980b333968ba-kube-api-access-ccx2s\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxqkv\" (UID: \"c6f95134-bc21-41b6-870d-980b333968ba\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.389708 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.389577 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f95134-bc21-41b6-870d-980b333968ba-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxqkv\" (UID: \"c6f95134-bc21-41b6-870d-980b333968ba\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.389708 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.389607 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f95134-bc21-41b6-870d-980b333968ba-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxqkv\" (UID: \"c6f95134-bc21-41b6-870d-980b333968ba\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.389708 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.389628 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f95134-bc21-41b6-870d-980b333968ba-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxqkv\" (UID: \"c6f95134-bc21-41b6-870d-980b333968ba\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.389708 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.389652 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f95134-bc21-41b6-870d-980b333968ba-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxqkv\" (UID: \"c6f95134-bc21-41b6-870d-980b333968ba\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.490867 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.490827 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccx2s\" (UniqueName: \"kubernetes.io/projected/c6f95134-bc21-41b6-870d-980b333968ba-kube-api-access-ccx2s\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxqkv\" (UID: \"c6f95134-bc21-41b6-870d-980b333968ba\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.491062 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.490873 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f95134-bc21-41b6-870d-980b333968ba-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxqkv\" (UID: \"c6f95134-bc21-41b6-870d-980b333968ba\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.491062 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.490922 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f95134-bc21-41b6-870d-980b333968ba-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxqkv\" (UID: \"c6f95134-bc21-41b6-870d-980b333968ba\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.491062 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.490951 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f95134-bc21-41b6-870d-980b333968ba-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxqkv\" (UID: \"c6f95134-bc21-41b6-870d-980b333968ba\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.491062 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.490975 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f95134-bc21-41b6-870d-980b333968ba-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxqkv\" (UID: \"c6f95134-bc21-41b6-870d-980b333968ba\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.491062 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.491025 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f95134-bc21-41b6-870d-980b333968ba-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxqkv\" (UID: \"c6f95134-bc21-41b6-870d-980b333968ba\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.491394 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.491368 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f95134-bc21-41b6-870d-980b333968ba-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxqkv\" (UID: \"c6f95134-bc21-41b6-870d-980b333968ba\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.491467 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.491416 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f95134-bc21-41b6-870d-980b333968ba-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxqkv\" (UID: \"c6f95134-bc21-41b6-870d-980b333968ba\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.491467 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.491438 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f95134-bc21-41b6-870d-980b333968ba-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxqkv\" (UID: \"c6f95134-bc21-41b6-870d-980b333968ba\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.493418 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.493395 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f95134-bc21-41b6-870d-980b333968ba-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxqkv\" (UID: \"c6f95134-bc21-41b6-870d-980b333968ba\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.493636 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.493617 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f95134-bc21-41b6-870d-980b333968ba-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxqkv\" (UID: \"c6f95134-bc21-41b6-870d-980b333968ba\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.503703 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.503675 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccx2s\" (UniqueName: \"kubernetes.io/projected/c6f95134-bc21-41b6-870d-980b333968ba-kube-api-access-ccx2s\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxqkv\" (UID: \"c6f95134-bc21-41b6-870d-980b333968ba\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.643429 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.643315 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:49:55.746891 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.746862 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:49:55.789680 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.789656 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv"] Jul 15 19:49:55.790975 ip-10-0-135-139 kubenswrapper[2554]: W0715 19:49:55.790945 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f95134_bc21_41b6_870d_980b333968ba.slice/crio-cc672265dcdb9e2ff0c03791ba3b3d2f003f59a88eee9f67f5730db166340537 WatchSource:0}: Error finding container cc672265dcdb9e2ff0c03791ba3b3d2f003f59a88eee9f67f5730db166340537: Status 404 returned error can't find the container with id cc672265dcdb9e2ff0c03791ba3b3d2f003f59a88eee9f67f5730db166340537 Jul 15 19:49:55.792398 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.792378 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:49:55.947246 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:55.947199 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:49:56.526835 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:56.526799 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" event={"ID":"c6f95134-bc21-41b6-870d-980b333968ba","Type":"ContainerStarted","Data":"cc672265dcdb9e2ff0c03791ba3b3d2f003f59a88eee9f67f5730db166340537"} Jul 15 19:49:57.533243 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:57.533211 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" event={"ID":"c6f95134-bc21-41b6-870d-980b333968ba","Type":"ContainerStarted","Data":"084d76504e268edcce7652ca13c5092140a87cb96fc9cd8a9bf74c242b224d01"} Jul 15 19:49:57.558805 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:49:57.558719 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" podStartSLOduration=1.493443862 podStartE2EDuration="2.558702961s" totalImagesPullingTime="1.065259099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:49:57.557195675 +0000 UTC m=+711.801702140" watchObservedRunningTime="2026-07-15 19:49:57.558702961 +0000 UTC m=+711.803209463" Jul 15 19:50:05.643547 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:50:05.643502 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:50:05.644105 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:50:05.643571 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:50:05.658516 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:50:05.658478 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:50:06.588649 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:50:06.588614 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxqkv" Jul 15 19:50:09.971251 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:50:09.971209 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:11.339778 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:50:11.339709 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:50:19.755431 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:50:19.755394 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:21.343161 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:50:21.343120 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:50:38.169258 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:50:38.169217 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:38.290449 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:50:38.290402 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:39.740215 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:50:39.740181 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:41.836227 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:50:41.836193 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:50:42.040517 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:50:42.040476 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:44.952919 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:50:44.952879 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:51.624974 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:50:51.624941 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:52:01.893234 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:52:01.893188 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:52:02.060254 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:52:02.060209 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:52:02.994761 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:52:02.994727 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:52:03.082296 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:52:03.082253 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:52:29.639596 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:52:29.639552 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:52:40.028385 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:52:40.028346 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:52:50.128306 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:52:50.128265 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:52:59.641510 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:52:59.641424 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:53:06.248457 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:53:06.248423 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 19:53:06.251652 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:53:06.251635 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 19:53:06.253081 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:53:06.253063 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 19:53:06.256173 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:53:06.256152 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 19:53:09.333333 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:53:09.333295 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:53:12.405274 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:53:12.405239 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:14.160536 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:53:14.160498 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:38.734079 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:53:38.734040 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:53:52.257824 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:53:52.257791 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:53:55.742605 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:53:55.742551 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:56.931477 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:53:56.931441 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:02.256477 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:54:02.256438 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:02.367037 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:54:02.367004 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:02.462861 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:54:02.462819 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:03.350085 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:54:03.345224 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:09.842411 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:54:09.842371 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:09.966381 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:54:09.966351 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:27.038848 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:54:27.038746 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:54:38.546478 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:54:38.546438 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:54:49.130576 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:54:49.130533 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:54:56.350408 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:54:56.350371 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:55:08.104289 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:55:08.104247 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:55:19.546183 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:55:19.546142 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:55:30.337458 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:55:30.337420 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:55:40.336701 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:55:40.336670 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:55:49.453388 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:55:49.453335 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:56:02.543484 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:56:02.543401 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:56:11.330846 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:56:11.330809 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:56:27.663780 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:56:27.663721 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:56:33.276650 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:56:33.276611 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:56:36.127813 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:56:36.127779 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:56:46.640153 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:56:46.640115 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:57:03.438249 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:57:03.438210 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:57:14.630429 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:57:14.630390 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:57:23.742414 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:57:23.742381 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:57:33.533779 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:57:33.533686 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:57:43.041605 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:57:43.041569 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:58:06.280942 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:58:06.280909 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 19:58:06.285607 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:58:06.285582 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 19:58:06.287031 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:58:06.287009 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 19:58:06.291245 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:58:06.291225 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 19:58:59.248213 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:58:59.248128 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:30.834875 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:59:30.834838 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:59:39.635040 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:59:39.635002 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 19:59:52.027892 ip-10-0-135-139 kubenswrapper[2554]: I0715 19:59:52.027852 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:00:00.142241 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:00.142198 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735760-kvmlk"] Jul 15 20:00:00.145629 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:00.145606 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735760-kvmlk" Jul 15 20:00:00.148443 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:00.148425 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tkj8f\"" Jul 15 20:00:00.154556 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:00.154534 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735760-kvmlk"] Jul 15 20:00:00.156279 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:00.156255 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735760-kvmlk" Jul 15 20:00:00.294124 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:00.294096 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735760-kvmlk"] Jul 15 20:00:00.295705 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:00:00.295680 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d95f85_4de7_4369_8881_4c281515cbb4.slice/crio-65a03c450de15018342ec777c0e283f5eed4139b20007c4a8a530657199180d1 WatchSource:0}: Error finding container 65a03c450de15018342ec777c0e283f5eed4139b20007c4a8a530657199180d1: Status 404 returned error can't find the container with id 65a03c450de15018342ec777c0e283f5eed4139b20007c4a8a530657199180d1 Jul 15 20:00:00.297551 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:00.297536 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 20:00:00.531939 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:00.531904 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735760-kvmlk" event={"ID":"f7d95f85-4de7-4369-8881-4c281515cbb4","Type":"ContainerStarted","Data":"65a03c450de15018342ec777c0e283f5eed4139b20007c4a8a530657199180d1"} Jul 15 20:00:03.439487 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:03.439447 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:00:07.569898 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:07.569859 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735760-kvmlk" event={"ID":"f7d95f85-4de7-4369-8881-4c281515cbb4","Type":"ContainerStarted","Data":"225af323319080fc2dd61e79f43cbc1264e4fe5e600f2d6214bdb4f798b97672"} Jul 15 20:00:07.608715 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:07.608665 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735760-kvmlk" podStartSLOduration=1.121854976 podStartE2EDuration="7.608645459s" totalImagesPullingTime="6.486790483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:00:07.608110687 +0000 UTC m=+1321.852617149" watchObservedRunningTime="2026-07-15 20:00:07.608645459 +0000 UTC m=+1321.853151920" Jul 15 20:00:08.582006 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:08.581967 2554 generic.go:356] "Generic (PLEG): container finished" podID="f7d95f85-4de7-4369-8881-4c281515cbb4" containerID="225af323319080fc2dd61e79f43cbc1264e4fe5e600f2d6214bdb4f798b97672" exitCode=0 Jul 15 20:00:08.582400 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:08.582051 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735760-kvmlk" event={"ID":"f7d95f85-4de7-4369-8881-4c281515cbb4","Type":"ContainerDied","Data":"225af323319080fc2dd61e79f43cbc1264e4fe5e600f2d6214bdb4f798b97672"} Jul 15 20:00:09.716183 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:09.716154 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735760-kvmlk" Jul 15 20:00:10.594822 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:10.594732 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735760-kvmlk" Jul 15 20:00:10.594822 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:10.594773 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735760-kvmlk" event={"ID":"f7d95f85-4de7-4369-8881-4c281515cbb4","Type":"ContainerDied","Data":"65a03c450de15018342ec777c0e283f5eed4139b20007c4a8a530657199180d1"} Jul 15 20:00:10.594822 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:10.594818 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65a03c450de15018342ec777c0e283f5eed4139b20007c4a8a530657199180d1" Jul 15 20:00:15.128546 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:15.128511 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:00:25.638545 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:25.638503 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:00:33.535062 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:33.535024 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:00:44.336032 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:44.335997 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:00:52.239844 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:00:52.239810 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:01:01.132049 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:01:01.132011 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:01:12.333607 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:01:12.333573 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:01:20.148965 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:01:20.148928 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:01:37.445622 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:01:37.445580 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:01:46.838853 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:01:46.838813 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:01:51.527258 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:01:51.527215 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:51.697624 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:01:51.697588 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:51.763380 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:01:51.763337 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:58.672017 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:01:58.671907 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:02:06.842168 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:02:06.842127 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:09.328671 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:02:09.328633 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:02:28.453946 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:02:28.453909 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:02:37.855788 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:02:37.855728 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:02:48.642166 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:02:48.642125 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:02:56.878051 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:02:56.878012 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:03:06.312889 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:03:06.312858 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 20:03:06.317486 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:03:06.317461 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 20:03:06.320286 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:03:06.320264 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 20:03:06.324655 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:03:06.324633 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 20:03:06.465510 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:03:06.465473 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:03:16.256250 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:03:16.256207 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:03:30.139464 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:03:30.139383 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:03:39.675121 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:03:39.675088 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:03:55.708965 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:03:55.708924 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:04:06.573926 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:04:06.573892 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:04:18.642926 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:04:18.642890 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:04:36.247058 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:04:36.247026 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:04:47.936642 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:04:47.936604 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:04:59.355252 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:04:59.355161 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:05:03.264678 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:03.264648 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:04.525360 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:04.525326 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:11.883415 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:11.883382 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:12.545110 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:12.545069 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:05:13.893205 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:13.893171 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:20.138826 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:20.138791 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:25.366802 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:25.366733 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:28.541403 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:28.541373 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:28.643724 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:28.643691 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:29.547527 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:29.547491 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:05:30.431031 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:30.430997 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:35.230564 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:35.230497 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:35.333272 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:35.333239 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:35.504583 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:35.504494 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:35.601451 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:35.601421 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:36.718958 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:36.718925 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:36.859456 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:36.859422 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:36.941494 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:36.941466 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96"] Jul 15 20:05:36.941814 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:36.941801 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7d95f85-4de7-4369-8881-4c281515cbb4" containerName="cleanup" Jul 15 20:05:36.941814 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:36.941815 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7d95f85-4de7-4369-8881-4c281515cbb4" containerName="cleanup" Jul 15 20:05:36.941945 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:36.941877 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7d95f85-4de7-4369-8881-4c281515cbb4" containerName="cleanup" Jul 15 20:05:36.943851 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:36.943836 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" Jul 15 20:05:36.946729 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:36.946705 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-10eed66e-serving-cert\"" Jul 15 20:05:36.947084 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:36.947068 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 20:05:36.947141 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:36.947085 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 20:05:36.947731 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:36.947719 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c5cgj\"" Jul 15 20:05:36.961612 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:36.961584 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96"] Jul 15 20:05:37.021869 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:37.021780 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38bdbb7-caf0-415a-9017-2b04c884285e-openshift-service-ca\") pod \"maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96\" (UID: \"a38bdbb7-caf0-415a-9017-2b04c884285e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" Jul 15 20:05:37.021869 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:37.021832 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a38bdbb7-caf0-415a-9017-2b04c884285e-maas-api-tls\") pod \"maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96\" (UID: \"a38bdbb7-caf0-415a-9017-2b04c884285e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" Jul 15 20:05:37.022045 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:37.021888 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n94vd\" (UniqueName: \"kubernetes.io/projected/a38bdbb7-caf0-415a-9017-2b04c884285e-kube-api-access-n94vd\") pod \"maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96\" (UID: \"a38bdbb7-caf0-415a-9017-2b04c884285e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" Jul 15 20:05:37.053369 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:37.053330 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:37.122919 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:37.122887 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38bdbb7-caf0-415a-9017-2b04c884285e-openshift-service-ca\") pod \"maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96\" (UID: \"a38bdbb7-caf0-415a-9017-2b04c884285e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" Jul 15 20:05:37.123196 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:37.122932 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a38bdbb7-caf0-415a-9017-2b04c884285e-maas-api-tls\") pod \"maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96\" (UID: \"a38bdbb7-caf0-415a-9017-2b04c884285e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" Jul 15 20:05:37.123196 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:37.122952 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n94vd\" (UniqueName: \"kubernetes.io/projected/a38bdbb7-caf0-415a-9017-2b04c884285e-kube-api-access-n94vd\") pod \"maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96\" (UID: \"a38bdbb7-caf0-415a-9017-2b04c884285e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" Jul 15 20:05:37.123768 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:37.123723 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38bdbb7-caf0-415a-9017-2b04c884285e-openshift-service-ca\") pod \"maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96\" (UID: \"a38bdbb7-caf0-415a-9017-2b04c884285e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" Jul 15 20:05:37.126416 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:37.126387 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a38bdbb7-caf0-415a-9017-2b04c884285e-maas-api-tls\") pod \"maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96\" (UID: \"a38bdbb7-caf0-415a-9017-2b04c884285e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" Jul 15 20:05:37.136838 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:37.136806 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n94vd\" (UniqueName: \"kubernetes.io/projected/a38bdbb7-caf0-415a-9017-2b04c884285e-kube-api-access-n94vd\") pod \"maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96\" (UID: \"a38bdbb7-caf0-415a-9017-2b04c884285e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" Jul 15 20:05:37.154626 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:37.154593 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:37.255759 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:37.255711 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" Jul 15 20:05:37.397124 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:37.397041 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96"] Jul 15 20:05:37.399433 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:05:37.399397 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38bdbb7_caf0_415a_9017_2b04c884285e.slice/crio-103a9dcfb9ff5404fc2757042bda6f1be58040ee3250a592ce9b3131e80d6eab WatchSource:0}: Error finding container 103a9dcfb9ff5404fc2757042bda6f1be58040ee3250a592ce9b3131e80d6eab: Status 404 returned error can't find the container with id 103a9dcfb9ff5404fc2757042bda6f1be58040ee3250a592ce9b3131e80d6eab Jul 15 20:05:37.401075 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:37.401053 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 20:05:38.244330 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:38.244294 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" event={"ID":"a38bdbb7-caf0-415a-9017-2b04c884285e","Type":"ContainerStarted","Data":"103a9dcfb9ff5404fc2757042bda6f1be58040ee3250a592ce9b3131e80d6eab"} Jul 15 20:05:39.035083 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:39.035032 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:39.101525 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:39.101482 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:40.077869 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:40.077841 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:40.256095 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:40.256063 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" event={"ID":"a38bdbb7-caf0-415a-9017-2b04c884285e","Type":"ContainerStarted","Data":"d58bf6777b71bd256c9ddf863f77138a8f5c97987a47c291c305bf5d1788f71f"} Jul 15 20:05:40.256265 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:40.256152 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" Jul 15 20:05:40.277414 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:40.277362 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" podStartSLOduration=1.985012399 podStartE2EDuration="4.277348416s" totalImagesPullingTime="2.292336017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:05:40.276733755 +0000 UTC m=+1654.521240217" watchObservedRunningTime="2026-07-15 20:05:40.277348416 +0000 UTC m=+1654.521854917" Jul 15 20:05:40.632887 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:40.632851 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:05:41.666309 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:41.666273 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb"] Jul 15 20:05:41.669394 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:41.669369 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb" Jul 15 20:05:41.673316 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:41.673297 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tkj8f\"" Jul 15 20:05:41.689039 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:41.689012 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb"] Jul 15 20:05:41.767567 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:41.767532 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/109bd395-23e8-4591-9d87-0d39deb9a7c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb\" (UID: \"109bd395-23e8-4591-9d87-0d39deb9a7c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb" Jul 15 20:05:41.868246 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:41.868211 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/109bd395-23e8-4591-9d87-0d39deb9a7c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb\" (UID: \"109bd395-23e8-4591-9d87-0d39deb9a7c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb" Jul 15 20:05:41.868953 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:41.868936 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/109bd395-23e8-4591-9d87-0d39deb9a7c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb\" (UID: \"109bd395-23e8-4591-9d87-0d39deb9a7c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb" Jul 15 20:05:41.979734 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:41.979694 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb" Jul 15 20:05:42.137552 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:42.137519 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb"] Jul 15 20:05:42.140894 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:05:42.140868 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod109bd395_23e8_4591_9d87_0d39deb9a7c1.slice/crio-cc7ce3519b2632b9fc1013c9b1041cef9ab5423739112396b9be59580f9a4797 WatchSource:0}: Error finding container cc7ce3519b2632b9fc1013c9b1041cef9ab5423739112396b9be59580f9a4797: Status 404 returned error can't find the container with id cc7ce3519b2632b9fc1013c9b1041cef9ab5423739112396b9be59580f9a4797 Jul 15 20:05:42.267382 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:42.267298 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb" event={"ID":"109bd395-23e8-4591-9d87-0d39deb9a7c1","Type":"ContainerStarted","Data":"cc7ce3519b2632b9fc1013c9b1041cef9ab5423739112396b9be59580f9a4797"} Jul 15 20:05:43.273670 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:43.273636 2554 generic.go:356] "Generic (PLEG): container finished" podID="109bd395-23e8-4591-9d87-0d39deb9a7c1" containerID="8eaff2c903ae0240e7352f023fa31dd2724bf479b4dec06e744d33844968a121" exitCode=7 Jul 15 20:05:43.274157 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:43.273705 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb" event={"ID":"109bd395-23e8-4591-9d87-0d39deb9a7c1","Type":"ContainerDied","Data":"8eaff2c903ae0240e7352f023fa31dd2724bf479b4dec06e744d33844968a121"} Jul 15 20:05:43.274157 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:43.273949 2554 scope.go:117] "RemoveContainer" containerID="8eaff2c903ae0240e7352f023fa31dd2724bf479b4dec06e744d33844968a121" Jul 15 20:05:44.280483 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:44.280443 2554 generic.go:356] "Generic (PLEG): container finished" podID="109bd395-23e8-4591-9d87-0d39deb9a7c1" containerID="f9cb20576679a83df9816d03154757fe9d7cecc04e280e40a240ac20096e1c9b" exitCode=7 Jul 15 20:05:44.281000 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:44.280536 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb" event={"ID":"109bd395-23e8-4591-9d87-0d39deb9a7c1","Type":"ContainerDied","Data":"f9cb20576679a83df9816d03154757fe9d7cecc04e280e40a240ac20096e1c9b"} Jul 15 20:05:44.281000 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:44.280607 2554 scope.go:117] "RemoveContainer" containerID="8eaff2c903ae0240e7352f023fa31dd2724bf479b4dec06e744d33844968a121" Jul 15 20:05:44.281000 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:44.280717 2554 scope.go:117] "RemoveContainer" containerID="f9cb20576679a83df9816d03154757fe9d7cecc04e280e40a240ac20096e1c9b" Jul 15 20:05:44.281000 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:05:44.280944 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb_odh-ai-gateway-infra(109bd395-23e8-4591-9d87-0d39deb9a7c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb" podUID="109bd395-23e8-4591-9d87-0d39deb9a7c1" Jul 15 20:05:45.287685 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:45.287658 2554 scope.go:117] "RemoveContainer" containerID="f9cb20576679a83df9816d03154757fe9d7cecc04e280e40a240ac20096e1c9b" Jul 15 20:05:45.288116 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:05:45.287866 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb_odh-ai-gateway-infra(109bd395-23e8-4591-9d87-0d39deb9a7c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb" podUID="109bd395-23e8-4591-9d87-0d39deb9a7c1" Jul 15 20:05:46.268185 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:46.268155 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" Jul 15 20:05:56.240235 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:56.240204 2554 scope.go:117] "RemoveContainer" containerID="f9cb20576679a83df9816d03154757fe9d7cecc04e280e40a240ac20096e1c9b" Jul 15 20:05:57.351577 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:57.351494 2554 generic.go:356] "Generic (PLEG): container finished" podID="109bd395-23e8-4591-9d87-0d39deb9a7c1" containerID="0da3e00e21a3a6fa8ec33cd7385b6779cf512d182a012462418cab67757b2acb" exitCode=0 Jul 15 20:05:57.351577 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:57.351555 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb" event={"ID":"109bd395-23e8-4591-9d87-0d39deb9a7c1","Type":"ContainerDied","Data":"0da3e00e21a3a6fa8ec33cd7385b6779cf512d182a012462418cab67757b2acb"} Jul 15 20:05:57.352053 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:57.351620 2554 scope.go:117] "RemoveContainer" containerID="f9cb20576679a83df9816d03154757fe9d7cecc04e280e40a240ac20096e1c9b" Jul 15 20:05:58.387061 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:58.387027 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb"] Jul 15 20:05:58.494553 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:58.494529 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb" Jul 15 20:05:58.619278 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:58.619197 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/109bd395-23e8-4591-9d87-0d39deb9a7c1-maas-api-service-ca\") pod \"109bd395-23e8-4591-9d87-0d39deb9a7c1\" (UID: \"109bd395-23e8-4591-9d87-0d39deb9a7c1\") " Jul 15 20:05:58.619564 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:58.619533 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/109bd395-23e8-4591-9d87-0d39deb9a7c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "109bd395-23e8-4591-9d87-0d39deb9a7c1" (UID: "109bd395-23e8-4591-9d87-0d39deb9a7c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:05:58.720440 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:58.720409 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/109bd395-23e8-4591-9d87-0d39deb9a7c1-maas-api-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:05:59.364661 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:59.364633 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb" Jul 15 20:05:59.364853 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:59.364671 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb" event={"ID":"109bd395-23e8-4591-9d87-0d39deb9a7c1","Type":"ContainerDied","Data":"cc7ce3519b2632b9fc1013c9b1041cef9ab5423739112396b9be59580f9a4797"} Jul 15 20:05:59.364853 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:59.364724 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc7ce3519b2632b9fc1013c9b1041cef9ab5423739112396b9be59580f9a4797" Jul 15 20:05:59.398619 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:59.398574 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb"] Jul 15 20:05:59.403490 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:05:59.403462 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9jwsb"] Jul 15 20:06:00.238398 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:00.238365 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="109bd395-23e8-4591-9d87-0d39deb9a7c1" path="/var/lib/kubelet/pods/109bd395-23e8-4591-9d87-0d39deb9a7c1/volumes" Jul 15 20:06:31.744131 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:31.744057 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9677l"] Jul 15 20:06:31.744570 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:31.744401 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="109bd395-23e8-4591-9d87-0d39deb9a7c1" containerName="revoke-keys" Jul 15 20:06:31.744570 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:31.744412 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="109bd395-23e8-4591-9d87-0d39deb9a7c1" containerName="revoke-keys" Jul 15 20:06:31.744570 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:31.744424 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="109bd395-23e8-4591-9d87-0d39deb9a7c1" containerName="revoke-keys" Jul 15 20:06:31.744570 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:31.744429 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="109bd395-23e8-4591-9d87-0d39deb9a7c1" containerName="revoke-keys" Jul 15 20:06:31.744570 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:31.744441 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="109bd395-23e8-4591-9d87-0d39deb9a7c1" containerName="revoke-keys" Jul 15 20:06:31.744570 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:31.744446 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="109bd395-23e8-4591-9d87-0d39deb9a7c1" containerName="revoke-keys" Jul 15 20:06:31.744570 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:31.744512 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="109bd395-23e8-4591-9d87-0d39deb9a7c1" containerName="revoke-keys" Jul 15 20:06:31.744570 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:31.744521 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="109bd395-23e8-4591-9d87-0d39deb9a7c1" containerName="revoke-keys" Jul 15 20:06:31.749604 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:31.749586 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9677l" Jul 15 20:06:31.752483 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:31.752459 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tkj8f\"" Jul 15 20:06:31.761876 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:31.761851 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9677l"] Jul 15 20:06:31.803816 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:31.803778 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeaf38aa-db6e-45be-bb8b-44b65c707670-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-10eed66e-9677l\" (UID: \"aeaf38aa-db6e-45be-bb8b-44b65c707670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9677l" Jul 15 20:06:31.904464 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:31.904427 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeaf38aa-db6e-45be-bb8b-44b65c707670-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-10eed66e-9677l\" (UID: \"aeaf38aa-db6e-45be-bb8b-44b65c707670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9677l" Jul 15 20:06:31.905086 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:31.905068 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeaf38aa-db6e-45be-bb8b-44b65c707670-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-10eed66e-9677l\" (UID: \"aeaf38aa-db6e-45be-bb8b-44b65c707670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9677l" Jul 15 20:06:32.060306 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:32.060202 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9677l" Jul 15 20:06:32.410779 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:32.407443 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9677l"] Jul 15 20:06:32.538587 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:32.538550 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9677l" event={"ID":"aeaf38aa-db6e-45be-bb8b-44b65c707670","Type":"ContainerStarted","Data":"68d09f16d60e016a4520a8d87249a2d699854cd48529da0edf9c2981f2e9ad69"} Jul 15 20:06:33.545638 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:33.545596 2554 generic.go:356] "Generic (PLEG): container finished" podID="aeaf38aa-db6e-45be-bb8b-44b65c707670" containerID="e6373bd1409cd03f1e477ca70da8adb963c9fc865e8f201455537de1addcf270" exitCode=0 Jul 15 20:06:33.546047 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:33.545642 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9677l" event={"ID":"aeaf38aa-db6e-45be-bb8b-44b65c707670","Type":"ContainerDied","Data":"e6373bd1409cd03f1e477ca70da8adb963c9fc865e8f201455537de1addcf270"} Jul 15 20:06:34.686068 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:34.686041 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9677l" Jul 15 20:06:34.829524 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:34.829437 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeaf38aa-db6e-45be-bb8b-44b65c707670-maas-api-service-ca\") pod \"aeaf38aa-db6e-45be-bb8b-44b65c707670\" (UID: \"aeaf38aa-db6e-45be-bb8b-44b65c707670\") " Jul 15 20:06:34.829838 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:34.829814 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeaf38aa-db6e-45be-bb8b-44b65c707670-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aeaf38aa-db6e-45be-bb8b-44b65c707670" (UID: "aeaf38aa-db6e-45be-bb8b-44b65c707670"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:06:34.930457 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:34.930417 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeaf38aa-db6e-45be-bb8b-44b65c707670-maas-api-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:06:35.558435 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:35.558403 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9677l" Jul 15 20:06:35.558435 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:35.558425 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-10eed66e-9677l" event={"ID":"aeaf38aa-db6e-45be-bb8b-44b65c707670","Type":"ContainerDied","Data":"68d09f16d60e016a4520a8d87249a2d699854cd48529da0edf9c2981f2e9ad69"} Jul 15 20:06:35.558738 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:35.558478 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68d09f16d60e016a4520a8d87249a2d699854cd48529da0edf9c2981f2e9ad69" Jul 15 20:06:42.408235 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:42.408197 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96"] Jul 15 20:06:42.408892 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:42.408534 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" podUID="a38bdbb7-caf0-415a-9017-2b04c884285e" containerName="maas-api" containerID="cri-o://d58bf6777b71bd256c9ddf863f77138a8f5c97987a47c291c305bf5d1788f71f" gracePeriod=30 Jul 15 20:06:42.464596 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:42.464564 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:06:42.597544 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:42.597512 2554 generic.go:356] "Generic (PLEG): container finished" podID="a38bdbb7-caf0-415a-9017-2b04c884285e" containerID="d58bf6777b71bd256c9ddf863f77138a8f5c97987a47c291c305bf5d1788f71f" exitCode=0 Jul 15 20:06:42.597687 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:42.597575 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" event={"ID":"a38bdbb7-caf0-415a-9017-2b04c884285e","Type":"ContainerDied","Data":"d58bf6777b71bd256c9ddf863f77138a8f5c97987a47c291c305bf5d1788f71f"} Jul 15 20:06:42.659295 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:42.659238 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" Jul 15 20:06:42.800731 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:42.800684 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n94vd\" (UniqueName: \"kubernetes.io/projected/a38bdbb7-caf0-415a-9017-2b04c884285e-kube-api-access-n94vd\") pod \"a38bdbb7-caf0-415a-9017-2b04c884285e\" (UID: \"a38bdbb7-caf0-415a-9017-2b04c884285e\") " Jul 15 20:06:42.800937 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:42.800794 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38bdbb7-caf0-415a-9017-2b04c884285e-openshift-service-ca\") pod \"a38bdbb7-caf0-415a-9017-2b04c884285e\" (UID: \"a38bdbb7-caf0-415a-9017-2b04c884285e\") " Jul 15 20:06:42.800937 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:42.800826 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a38bdbb7-caf0-415a-9017-2b04c884285e-maas-api-tls\") pod \"a38bdbb7-caf0-415a-9017-2b04c884285e\" (UID: \"a38bdbb7-caf0-415a-9017-2b04c884285e\") " Jul 15 20:06:42.801213 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:42.801179 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38bdbb7-caf0-415a-9017-2b04c884285e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a38bdbb7-caf0-415a-9017-2b04c884285e" (UID: "a38bdbb7-caf0-415a-9017-2b04c884285e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:06:42.803222 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:42.803188 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a38bdbb7-caf0-415a-9017-2b04c884285e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a38bdbb7-caf0-415a-9017-2b04c884285e" (UID: "a38bdbb7-caf0-415a-9017-2b04c884285e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:06:42.803222 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:42.803216 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38bdbb7-caf0-415a-9017-2b04c884285e-kube-api-access-n94vd" (OuterVolumeSpecName: "kube-api-access-n94vd") pod "a38bdbb7-caf0-415a-9017-2b04c884285e" (UID: "a38bdbb7-caf0-415a-9017-2b04c884285e"). InnerVolumeSpecName "kube-api-access-n94vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:06:42.901828 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:42.901780 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a38bdbb7-caf0-415a-9017-2b04c884285e-maas-api-tls\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:06:42.901828 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:42.901822 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n94vd\" (UniqueName: \"kubernetes.io/projected/a38bdbb7-caf0-415a-9017-2b04c884285e-kube-api-access-n94vd\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:06:42.901828 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:42.901832 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38bdbb7-caf0-415a-9017-2b04c884285e-openshift-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:06:43.605014 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:43.604985 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" Jul 15 20:06:43.605438 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:43.605020 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96" event={"ID":"a38bdbb7-caf0-415a-9017-2b04c884285e","Type":"ContainerDied","Data":"103a9dcfb9ff5404fc2757042bda6f1be58040ee3250a592ce9b3131e80d6eab"} Jul 15 20:06:43.605438 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:43.605068 2554 scope.go:117] "RemoveContainer" containerID="d58bf6777b71bd256c9ddf863f77138a8f5c97987a47c291c305bf5d1788f71f" Jul 15 20:06:43.631420 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:43.631391 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96"] Jul 15 20:06:43.635658 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:43.635637 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10eed66e-d8cdd6d5c-72r96"] Jul 15 20:06:44.239664 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:44.239629 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a38bdbb7-caf0-415a-9017-2b04c884285e" path="/var/lib/kubelet/pods/a38bdbb7-caf0-415a-9017-2b04c884285e/volumes" Jul 15 20:06:45.038692 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:45.038655 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:06:46.984241 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:46.984212 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:06:53.500139 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:53.500104 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:06:53.681644 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:53.681609 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:06:57.053451 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.053417 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:06:57.458630 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.458592 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:06:57.563675 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.563604 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:06:57.633695 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.633663 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2"] Jul 15 20:06:57.634120 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.634102 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a38bdbb7-caf0-415a-9017-2b04c884285e" containerName="maas-api" Jul 15 20:06:57.634210 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.634122 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a38bdbb7-caf0-415a-9017-2b04c884285e" containerName="maas-api" Jul 15 20:06:57.634210 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.634142 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeaf38aa-db6e-45be-bb8b-44b65c707670" containerName="revoke-keys" Jul 15 20:06:57.634210 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.634149 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeaf38aa-db6e-45be-bb8b-44b65c707670" containerName="revoke-keys" Jul 15 20:06:57.634373 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.634232 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeaf38aa-db6e-45be-bb8b-44b65c707670" containerName="revoke-keys" Jul 15 20:06:57.634373 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.634246 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="109bd395-23e8-4591-9d87-0d39deb9a7c1" containerName="revoke-keys" Jul 15 20:06:57.634373 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.634256 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a38bdbb7-caf0-415a-9017-2b04c884285e" containerName="maas-api" Jul 15 20:06:57.638824 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.638802 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.643176 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.642697 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-73e5b6fd-gw-openshift-default-dockercfg-d4b2q\"" Jul 15 20:06:57.660439 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.660406 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2"] Jul 15 20:06:57.696344 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.696304 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:06:57.737902 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.737808 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-credential-socket\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.738181 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.738145 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-envoy\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.738349 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.738200 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-podinfo\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.738349 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.738335 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istiod-ca-cert\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.738578 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.738434 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-token\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.738578 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.738469 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-data\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.738578 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.738504 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brpdm\" (UniqueName: \"kubernetes.io/projected/bfb2c523-a0dc-4e92-aaed-534f1b611dae-kube-api-access-brpdm\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.738926 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.738591 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-workload-socket\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.738926 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.738696 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-workload-certs\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.840067 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.840031 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-data\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.840269 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.840078 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brpdm\" (UniqueName: \"kubernetes.io/projected/bfb2c523-a0dc-4e92-aaed-534f1b611dae-kube-api-access-brpdm\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.840269 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.840118 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-workload-socket\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.840269 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.840151 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-workload-certs\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.840269 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.840196 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-credential-socket\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.840269 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.840218 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-envoy\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.840269 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.840243 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-podinfo\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.840269 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.840267 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istiod-ca-cert\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.840633 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.840322 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-token\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.840633 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.840522 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-data\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.840633 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.840562 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-workload-socket\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.840633 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.840608 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-credential-socket\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.840633 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.840626 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-workload-certs\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.841069 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.841050 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istiod-ca-cert\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.842707 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.842685 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-envoy\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.842865 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.842846 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-podinfo\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.851361 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.851338 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-token\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.851487 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.851368 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brpdm\" (UniqueName: \"kubernetes.io/projected/bfb2c523-a0dc-4e92-aaed-534f1b611dae-kube-api-access-brpdm\") pod \"e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:57.949675 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:57.949631 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:06:58.092644 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:58.092608 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2"] Jul 15 20:06:58.093913 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:06:58.093881 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb2c523_a0dc_4e92_aaed_534f1b611dae.slice/crio-d0d76ad0f7bcd1c30613eabfa032894e16f76c9dae2d611467e42e58cf29e196 WatchSource:0}: Error finding container d0d76ad0f7bcd1c30613eabfa032894e16f76c9dae2d611467e42e58cf29e196: Status 404 returned error can't find the container with id d0d76ad0f7bcd1c30613eabfa032894e16f76c9dae2d611467e42e58cf29e196 Jul 15 20:06:58.690853 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:58.690818 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" event={"ID":"bfb2c523-a0dc-4e92-aaed-534f1b611dae","Type":"ContainerStarted","Data":"d0d76ad0f7bcd1c30613eabfa032894e16f76c9dae2d611467e42e58cf29e196"} Jul 15 20:06:58.977960 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:58.977864 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:06:59.126143 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:59.126115 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck"] Jul 15 20:06:59.130005 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:59.129982 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" Jul 15 20:06:59.133952 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:59.133919 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c5cgj\"" Jul 15 20:06:59.134207 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:59.134185 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 20:06:59.134954 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:59.134933 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-73e5b6fd-serving-cert\"" Jul 15 20:06:59.135078 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:59.134972 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 20:06:59.145979 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:59.145954 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck"] Jul 15 20:06:59.157584 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:59.157553 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:06:59.253897 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:59.253813 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e155341-bbcd-4dc8-b80a-4c208b9c5374-maas-api-tls\") pod \"maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck\" (UID: \"4e155341-bbcd-4dc8-b80a-4c208b9c5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" Jul 15 20:06:59.254067 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:59.253996 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e155341-bbcd-4dc8-b80a-4c208b9c5374-openshift-service-ca\") pod \"maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck\" (UID: \"4e155341-bbcd-4dc8-b80a-4c208b9c5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" Jul 15 20:06:59.254067 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:59.254024 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfhsc\" (UniqueName: \"kubernetes.io/projected/4e155341-bbcd-4dc8-b80a-4c208b9c5374-kube-api-access-xfhsc\") pod \"maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck\" (UID: \"4e155341-bbcd-4dc8-b80a-4c208b9c5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" Jul 15 20:06:59.355268 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:59.355223 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e155341-bbcd-4dc8-b80a-4c208b9c5374-openshift-service-ca\") pod \"maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck\" (UID: \"4e155341-bbcd-4dc8-b80a-4c208b9c5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" Jul 15 20:06:59.355446 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:59.355277 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfhsc\" (UniqueName: \"kubernetes.io/projected/4e155341-bbcd-4dc8-b80a-4c208b9c5374-kube-api-access-xfhsc\") pod \"maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck\" (UID: \"4e155341-bbcd-4dc8-b80a-4c208b9c5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" Jul 15 20:06:59.355446 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:59.355358 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e155341-bbcd-4dc8-b80a-4c208b9c5374-maas-api-tls\") pod \"maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck\" (UID: \"4e155341-bbcd-4dc8-b80a-4c208b9c5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" Jul 15 20:06:59.355660 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:06:59.355637 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-serving-cert: secret "maas-api-e2e-migrate-73e5b6fd-serving-cert" not found Jul 15 20:06:59.355781 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:06:59.355714 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e155341-bbcd-4dc8-b80a-4c208b9c5374-maas-api-tls podName:4e155341-bbcd-4dc8-b80a-4c208b9c5374 nodeName:}" failed. No retries permitted until 2026-07-15 20:06:59.855691276 +0000 UTC m=+1734.100197728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e155341-bbcd-4dc8-b80a-4c208b9c5374-maas-api-tls") pod "maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" (UID: "4e155341-bbcd-4dc8-b80a-4c208b9c5374") : secret "maas-api-e2e-migrate-73e5b6fd-serving-cert" not found Jul 15 20:06:59.356077 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:59.356029 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e155341-bbcd-4dc8-b80a-4c208b9c5374-openshift-service-ca\") pod \"maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck\" (UID: \"4e155341-bbcd-4dc8-b80a-4c208b9c5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" Jul 15 20:06:59.369459 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:59.369414 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfhsc\" (UniqueName: \"kubernetes.io/projected/4e155341-bbcd-4dc8-b80a-4c208b9c5374-kube-api-access-xfhsc\") pod \"maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck\" (UID: \"4e155341-bbcd-4dc8-b80a-4c208b9c5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" Jul 15 20:06:59.685782 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:59.685717 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:06:59.785869 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:59.785830 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:06:59.860149 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:06:59.860110 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e155341-bbcd-4dc8-b80a-4c208b9c5374-maas-api-tls\") pod \"maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck\" (UID: \"4e155341-bbcd-4dc8-b80a-4c208b9c5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" Jul 15 20:06:59.860378 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:06:59.860302 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-serving-cert: secret "maas-api-e2e-migrate-73e5b6fd-serving-cert" not found Jul 15 20:06:59.860457 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:06:59.860409 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e155341-bbcd-4dc8-b80a-4c208b9c5374-maas-api-tls podName:4e155341-bbcd-4dc8-b80a-4c208b9c5374 nodeName:}" failed. No retries permitted until 2026-07-15 20:07:00.860376568 +0000 UTC m=+1735.104883020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e155341-bbcd-4dc8-b80a-4c208b9c5374-maas-api-tls") pod "maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" (UID: "4e155341-bbcd-4dc8-b80a-4c208b9c5374") : secret "maas-api-e2e-migrate-73e5b6fd-serving-cert" not found Jul 15 20:07:00.049582 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:00.049488 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:00.713927 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:00.713892 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:07:00.714191 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:00.713964 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:07:00.714191 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:00.713995 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:07:00.869054 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:00.869008 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e155341-bbcd-4dc8-b80a-4c208b9c5374-maas-api-tls\") pod \"maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck\" (UID: \"4e155341-bbcd-4dc8-b80a-4c208b9c5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" Jul 15 20:07:00.871894 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:00.871862 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e155341-bbcd-4dc8-b80a-4c208b9c5374-maas-api-tls\") pod \"maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck\" (UID: \"4e155341-bbcd-4dc8-b80a-4c208b9c5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" Jul 15 20:07:00.943858 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:00.943815 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" Jul 15 20:07:01.084247 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:01.084222 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck"] Jul 15 20:07:01.085920 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:07:01.085883 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e155341_bbcd_4dc8_b80a_4c208b9c5374.slice/crio-14cff57b1624be8c749afa8e4ae2edd7a43693cd1d8fb8ab8d8d8064a2a1a3b9 WatchSource:0}: Error finding container 14cff57b1624be8c749afa8e4ae2edd7a43693cd1d8fb8ab8d8d8064a2a1a3b9: Status 404 returned error can't find the container with id 14cff57b1624be8c749afa8e4ae2edd7a43693cd1d8fb8ab8d8d8064a2a1a3b9 Jul 15 20:07:01.710064 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:01.710027 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" event={"ID":"bfb2c523-a0dc-4e92-aaed-534f1b611dae","Type":"ContainerStarted","Data":"45f9b1c1d47812fe45d340b5d58e3f6ccd0602aef2b35ab931fb632c2bebe597"} Jul 15 20:07:01.711575 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:01.711549 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" event={"ID":"4e155341-bbcd-4dc8-b80a-4c208b9c5374","Type":"ContainerStarted","Data":"52b54274ee05a81d4e2a9dbffdd08aa324433369776a31982019a5d7e936ac7a"} Jul 15 20:07:01.711681 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:01.711588 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" event={"ID":"4e155341-bbcd-4dc8-b80a-4c208b9c5374","Type":"ContainerStarted","Data":"14cff57b1624be8c749afa8e4ae2edd7a43693cd1d8fb8ab8d8d8064a2a1a3b9"} Jul 15 20:07:01.711681 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:01.711627 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" Jul 15 20:07:01.739400 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:01.739351 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" podStartSLOduration=2.121559302 podStartE2EDuration="4.739335063s" totalImagesPullingTime="2.617775761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:06:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:07:01.738371983 +0000 UTC m=+1735.982878444" watchObservedRunningTime="2026-07-15 20:07:01.739335063 +0000 UTC m=+1735.983841551" Jul 15 20:07:01.769415 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:01.769360 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" podStartSLOduration=2.277131984 podStartE2EDuration="2.7693425s" totalImagesPullingTime="492.210516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:07:01.767655848 +0000 UTC m=+1736.012162310" watchObservedRunningTime="2026-07-15 20:07:01.7693425 +0000 UTC m=+1736.013848952" Jul 15 20:07:01.950258 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:01.950219 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:07:01.955366 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:01.955338 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:07:02.718402 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:02.718370 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:07:02.719244 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:02.719226 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:07:03.412408 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:03.412376 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr"] Jul 15 20:07:03.415930 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:03.415913 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr" Jul 15 20:07:03.418953 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:03.418936 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tkj8f\"" Jul 15 20:07:03.427843 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:03.427822 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr"] Jul 15 20:07:03.485366 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:03.485325 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr\" (UID: \"9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr" Jul 15 20:07:03.586032 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:03.585989 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr\" (UID: \"9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr" Jul 15 20:07:03.586661 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:03.586640 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr\" (UID: \"9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr" Jul 15 20:07:03.728037 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:03.728007 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr" Jul 15 20:07:03.795574 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:03.795548 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:03.884655 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:03.884630 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr"] Jul 15 20:07:03.887061 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:07:03.887022 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a64ca88_0ff7_4608_9b0b_bf054bd3ce4b.slice/crio-582dae55733a73a0a7a5fc5b80abf5611e3acb9ce94ee0cc0d25e76d5597be45 WatchSource:0}: Error finding container 582dae55733a73a0a7a5fc5b80abf5611e3acb9ce94ee0cc0d25e76d5597be45: Status 404 returned error can't find the container with id 582dae55733a73a0a7a5fc5b80abf5611e3acb9ce94ee0cc0d25e76d5597be45 Jul 15 20:07:03.896435 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:03.896404 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:04.732487 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:04.732455 2554 generic.go:356] "Generic (PLEG): container finished" podID="9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b" containerID="1e6d48cd75d2782164f3094dfd45f3ea4eb5a99e3a80acc00cb96dc075450c55" exitCode=7 Jul 15 20:07:04.732891 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:04.732541 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr" event={"ID":"9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b","Type":"ContainerDied","Data":"1e6d48cd75d2782164f3094dfd45f3ea4eb5a99e3a80acc00cb96dc075450c55"} Jul 15 20:07:04.732891 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:04.732600 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr" event={"ID":"9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b","Type":"ContainerStarted","Data":"582dae55733a73a0a7a5fc5b80abf5611e3acb9ce94ee0cc0d25e76d5597be45"} Jul 15 20:07:04.732891 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:04.732713 2554 scope.go:117] "RemoveContainer" containerID="1e6d48cd75d2782164f3094dfd45f3ea4eb5a99e3a80acc00cb96dc075450c55" Jul 15 20:07:05.739958 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:05.739867 2554 generic.go:356] "Generic (PLEG): container finished" podID="9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b" containerID="6664d36f78d785a3ac24e0336ee11633caa81abfe8f9b2e873725406c5da7ce0" exitCode=7 Jul 15 20:07:05.740335 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:05.739954 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr" event={"ID":"9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b","Type":"ContainerDied","Data":"6664d36f78d785a3ac24e0336ee11633caa81abfe8f9b2e873725406c5da7ce0"} Jul 15 20:07:05.740335 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:05.740020 2554 scope.go:117] "RemoveContainer" containerID="1e6d48cd75d2782164f3094dfd45f3ea4eb5a99e3a80acc00cb96dc075450c55" Jul 15 20:07:05.740335 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:05.740272 2554 scope.go:117] "RemoveContainer" containerID="6664d36f78d785a3ac24e0336ee11633caa81abfe8f9b2e873725406c5da7ce0" Jul 15 20:07:05.740511 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:07:05.740491 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr_odh-ai-gateway-infra(9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr" podUID="9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b" Jul 15 20:07:06.027943 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:06.027854 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:07:06.745992 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:06.745963 2554 scope.go:117] "RemoveContainer" containerID="6664d36f78d785a3ac24e0336ee11633caa81abfe8f9b2e873725406c5da7ce0" Jul 15 20:07:06.746379 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:07:06.746174 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr_odh-ai-gateway-infra(9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr" podUID="9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b" Jul 15 20:07:07.149000 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:07.148903 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:07.732089 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:07.732049 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" Jul 15 20:07:14.791420 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:14.791382 2554 generic.go:356] "Generic (PLEG): container finished" podID="bfb2c523-a0dc-4e92-aaed-534f1b611dae" containerID="45f9b1c1d47812fe45d340b5d58e3f6ccd0602aef2b35ab931fb632c2bebe597" exitCode=0 Jul 15 20:07:14.791884 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:14.791457 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" event={"ID":"bfb2c523-a0dc-4e92-aaed-534f1b611dae","Type":"ContainerDied","Data":"45f9b1c1d47812fe45d340b5d58e3f6ccd0602aef2b35ab931fb632c2bebe597"} Jul 15 20:07:14.791884 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:14.791846 2554 scope.go:117] "RemoveContainer" containerID="45f9b1c1d47812fe45d340b5d58e3f6ccd0602aef2b35ab931fb632c2bebe597" Jul 15 20:07:14.792807 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:14.792783 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:07:14.792884 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:14.792853 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:07:14.792969 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:14.792886 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:07:15.798344 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:15.798306 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" event={"ID":"bfb2c523-a0dc-4e92-aaed-534f1b611dae","Type":"ContainerStarted","Data":"c5da4df297dba4050861853307a4fc762c96b91fc2a5900303cea5f8aac93cc8"} Jul 15 20:07:15.950235 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:15.950190 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:07:15.951304 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:15.951274 2554 patch_prober.go:28] interesting pod/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 15 20:07:15.951444 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:15.951339 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" podUID="bfb2c523-a0dc-4e92-aaed-534f1b611dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 15 20:07:16.950568 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:16.950532 2554 patch_prober.go:28] interesting pod/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 15 20:07:16.951067 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:16.950594 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" podUID="bfb2c523-a0dc-4e92-aaed-534f1b611dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 15 20:07:17.718452 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:17.718413 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:07:17.950199 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:17.950156 2554 patch_prober.go:28] interesting pod/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 15 20:07:17.950402 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:17.950235 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" podUID="bfb2c523-a0dc-4e92-aaed-534f1b611dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 15 20:07:18.950936 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:18.950898 2554 patch_prober.go:28] interesting pod/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 15 20:07:18.951319 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:18.950952 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" podUID="bfb2c523-a0dc-4e92-aaed-534f1b611dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 15 20:07:19.233395 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:19.233299 2554 scope.go:117] "RemoveContainer" containerID="6664d36f78d785a3ac24e0336ee11633caa81abfe8f9b2e873725406c5da7ce0" Jul 15 20:07:19.822350 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:19.822314 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr" event={"ID":"9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b","Type":"ContainerStarted","Data":"70554b713492c0c52c6aa39cabcbe4f987814376fab661787b9f4f7af25d233c"} Jul 15 20:07:19.844079 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:19.844028 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr" podStartSLOduration=16.844012909 podStartE2EDuration="16.844012909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:07:19.841775639 +0000 UTC m=+1754.086282097" watchObservedRunningTime="2026-07-15 20:07:19.844012909 +0000 UTC m=+1754.088519371" Jul 15 20:07:19.954368 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:19.954337 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:07:19.955283 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:19.955263 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:07:20.829942 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:20.829903 2554 generic.go:356] "Generic (PLEG): container finished" podID="9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b" containerID="70554b713492c0c52c6aa39cabcbe4f987814376fab661787b9f4f7af25d233c" exitCode=0 Jul 15 20:07:20.830118 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:20.829975 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr" event={"ID":"9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b","Type":"ContainerDied","Data":"70554b713492c0c52c6aa39cabcbe4f987814376fab661787b9f4f7af25d233c"} Jul 15 20:07:20.830118 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:20.830065 2554 scope.go:117] "RemoveContainer" containerID="6664d36f78d785a3ac24e0336ee11633caa81abfe8f9b2e873725406c5da7ce0" Jul 15 20:07:20.857775 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:20.857715 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr"] Jul 15 20:07:21.974708 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:21.974683 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr" Jul 15 20:07:22.048578 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:22.048547 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b-maas-api-service-ca\") pod \"9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b\" (UID: \"9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b\") " Jul 15 20:07:22.048950 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:22.048929 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b" (UID: "9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:07:22.150141 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:22.150057 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b-maas-api-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:07:22.843769 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:22.843718 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr" event={"ID":"9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b","Type":"ContainerDied","Data":"582dae55733a73a0a7a5fc5b80abf5611e3acb9ce94ee0cc0d25e76d5597be45"} Jul 15 20:07:22.843942 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:22.843796 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="582dae55733a73a0a7a5fc5b80abf5611e3acb9ce94ee0cc0d25e76d5597be45" Jul 15 20:07:22.843942 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:22.843732 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr" Jul 15 20:07:22.882068 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:22.882038 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr"] Jul 15 20:07:22.884139 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:22.884118 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kxbxr"] Jul 15 20:07:24.238872 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:24.238840 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b" path="/var/lib/kubelet/pods/9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b/volumes" Jul 15 20:07:53.429948 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:53.429907 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kfbrk"] Jul 15 20:07:53.430581 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:53.430429 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b" containerName="revoke-keys" Jul 15 20:07:53.430581 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:53.430450 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b" containerName="revoke-keys" Jul 15 20:07:53.430581 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:53.430498 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b" containerName="revoke-keys" Jul 15 20:07:53.430581 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:53.430507 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b" containerName="revoke-keys" Jul 15 20:07:53.430844 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:53.430588 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b" containerName="revoke-keys" Jul 15 20:07:53.430844 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:53.430599 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b" containerName="revoke-keys" Jul 15 20:07:53.433858 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:53.433834 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kfbrk" Jul 15 20:07:53.436609 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:53.436587 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tkj8f\"" Jul 15 20:07:53.442594 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:53.442567 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kfbrk"] Jul 15 20:07:53.525740 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:53.525694 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a9717a-2f91-48a3-b207-4204901a8e69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-73e5b6fd-kfbrk\" (UID: \"85a9717a-2f91-48a3-b207-4204901a8e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kfbrk" Jul 15 20:07:53.626655 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:53.626604 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a9717a-2f91-48a3-b207-4204901a8e69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-73e5b6fd-kfbrk\" (UID: \"85a9717a-2f91-48a3-b207-4204901a8e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kfbrk" Jul 15 20:07:53.627293 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:53.627270 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a9717a-2f91-48a3-b207-4204901a8e69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-73e5b6fd-kfbrk\" (UID: \"85a9717a-2f91-48a3-b207-4204901a8e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kfbrk" Jul 15 20:07:53.746042 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:53.745951 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kfbrk" Jul 15 20:07:53.895536 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:53.895509 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kfbrk"] Jul 15 20:07:53.897464 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:07:53.897434 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a9717a_2f91_48a3_b207_4204901a8e69.slice/crio-7595d2665ecb95684154529494ba7fca9946d49f6876281a74622a6e48b7126f WatchSource:0}: Error finding container 7595d2665ecb95684154529494ba7fca9946d49f6876281a74622a6e48b7126f: Status 404 returned error can't find the container with id 7595d2665ecb95684154529494ba7fca9946d49f6876281a74622a6e48b7126f Jul 15 20:07:54.006665 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:54.006579 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kfbrk" event={"ID":"85a9717a-2f91-48a3-b207-4204901a8e69","Type":"ContainerStarted","Data":"7595d2665ecb95684154529494ba7fca9946d49f6876281a74622a6e48b7126f"} Jul 15 20:07:55.013268 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:55.013229 2554 generic.go:356] "Generic (PLEG): container finished" podID="85a9717a-2f91-48a3-b207-4204901a8e69" containerID="c4b874eb008c6458af13bb03861db407018fe1a4549764591ac9972f74394c86" exitCode=0 Jul 15 20:07:55.013704 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:55.013317 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kfbrk" event={"ID":"85a9717a-2f91-48a3-b207-4204901a8e69","Type":"ContainerDied","Data":"c4b874eb008c6458af13bb03861db407018fe1a4549764591ac9972f74394c86"} Jul 15 20:07:56.158047 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:56.158019 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kfbrk" Jul 15 20:07:56.250615 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:56.250577 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a9717a-2f91-48a3-b207-4204901a8e69-maas-api-service-ca\") pod \"85a9717a-2f91-48a3-b207-4204901a8e69\" (UID: \"85a9717a-2f91-48a3-b207-4204901a8e69\") " Jul 15 20:07:56.250978 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:56.250952 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a9717a-2f91-48a3-b207-4204901a8e69-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "85a9717a-2f91-48a3-b207-4204901a8e69" (UID: "85a9717a-2f91-48a3-b207-4204901a8e69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:07:56.351934 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:56.351834 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a9717a-2f91-48a3-b207-4204901a8e69-maas-api-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:07:57.025380 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:57.025341 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kfbrk" event={"ID":"85a9717a-2f91-48a3-b207-4204901a8e69","Type":"ContainerDied","Data":"7595d2665ecb95684154529494ba7fca9946d49f6876281a74622a6e48b7126f"} Jul 15 20:07:57.025380 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:57.025364 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73e5b6fd-kfbrk" Jul 15 20:07:57.025597 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:07:57.025394 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7595d2665ecb95684154529494ba7fca9946d49f6876281a74622a6e48b7126f" Jul 15 20:08:03.512128 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:03.512095 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2"] Jul 15 20:08:03.512668 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:03.512301 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" podUID="bfb2c523-a0dc-4e92-aaed-534f1b611dae" containerName="istio-proxy" containerID="cri-o://c5da4df297dba4050861853307a4fc762c96b91fc2a5900303cea5f8aac93cc8" gracePeriod=30 Jul 15 20:08:03.545127 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:03.545095 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:03.645397 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:03.645360 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:03.947967 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:03.947934 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck"] Jul 15 20:08:03.948260 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:03.948221 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" podUID="4e155341-bbcd-4dc8-b80a-4c208b9c5374" containerName="maas-api" containerID="cri-o://52b54274ee05a81d4e2a9dbffdd08aa324433369776a31982019a5d7e936ac7a" gracePeriod=30 Jul 15 20:08:03.981482 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:03.981453 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:04.065623 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:04.065593 2554 generic.go:356] "Generic (PLEG): container finished" podID="4e155341-bbcd-4dc8-b80a-4c208b9c5374" containerID="52b54274ee05a81d4e2a9dbffdd08aa324433369776a31982019a5d7e936ac7a" exitCode=0 Jul 15 20:08:04.065807 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:04.065671 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" event={"ID":"4e155341-bbcd-4dc8-b80a-4c208b9c5374","Type":"ContainerDied","Data":"52b54274ee05a81d4e2a9dbffdd08aa324433369776a31982019a5d7e936ac7a"} Jul 15 20:08:04.211999 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:04.211937 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" Jul 15 20:08:04.217197 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:04.217178 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e155341-bbcd-4dc8-b80a-4c208b9c5374-maas-api-tls\") pod \"4e155341-bbcd-4dc8-b80a-4c208b9c5374\" (UID: \"4e155341-bbcd-4dc8-b80a-4c208b9c5374\") " Jul 15 20:08:04.217278 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:04.217219 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfhsc\" (UniqueName: \"kubernetes.io/projected/4e155341-bbcd-4dc8-b80a-4c208b9c5374-kube-api-access-xfhsc\") pod \"4e155341-bbcd-4dc8-b80a-4c208b9c5374\" (UID: \"4e155341-bbcd-4dc8-b80a-4c208b9c5374\") " Jul 15 20:08:04.217278 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:04.217248 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e155341-bbcd-4dc8-b80a-4c208b9c5374-openshift-service-ca\") pod \"4e155341-bbcd-4dc8-b80a-4c208b9c5374\" (UID: \"4e155341-bbcd-4dc8-b80a-4c208b9c5374\") " Jul 15 20:08:04.217673 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:04.217649 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e155341-bbcd-4dc8-b80a-4c208b9c5374-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4e155341-bbcd-4dc8-b80a-4c208b9c5374" (UID: "4e155341-bbcd-4dc8-b80a-4c208b9c5374"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:08:04.219352 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:04.219321 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e155341-bbcd-4dc8-b80a-4c208b9c5374-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4e155341-bbcd-4dc8-b80a-4c208b9c5374" (UID: "4e155341-bbcd-4dc8-b80a-4c208b9c5374"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:08:04.219446 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:04.219362 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e155341-bbcd-4dc8-b80a-4c208b9c5374-kube-api-access-xfhsc" (OuterVolumeSpecName: "kube-api-access-xfhsc") pod "4e155341-bbcd-4dc8-b80a-4c208b9c5374" (UID: "4e155341-bbcd-4dc8-b80a-4c208b9c5374"). InnerVolumeSpecName "kube-api-access-xfhsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:08:04.318490 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:04.318449 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e155341-bbcd-4dc8-b80a-4c208b9c5374-maas-api-tls\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:08:04.318490 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:04.318486 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfhsc\" (UniqueName: \"kubernetes.io/projected/4e155341-bbcd-4dc8-b80a-4c208b9c5374-kube-api-access-xfhsc\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:08:04.318490 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:04.318497 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e155341-bbcd-4dc8-b80a-4c208b9c5374-openshift-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:08:04.957738 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:04.957700 2554 patch_prober.go:28] interesting pod/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 20:08:04.958178 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:04.957799 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" podUID="bfb2c523-a0dc-4e92-aaed-534f1b611dae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 20:08:05.073385 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:05.073349 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" Jul 15 20:08:05.073555 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:05.073356 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck" event={"ID":"4e155341-bbcd-4dc8-b80a-4c208b9c5374","Type":"ContainerDied","Data":"14cff57b1624be8c749afa8e4ae2edd7a43693cd1d8fb8ab8d8d8064a2a1a3b9"} Jul 15 20:08:05.073555 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:05.073487 2554 scope.go:117] "RemoveContainer" containerID="52b54274ee05a81d4e2a9dbffdd08aa324433369776a31982019a5d7e936ac7a" Jul 15 20:08:05.092433 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:05.092404 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck"] Jul 15 20:08:05.099383 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:05.099315 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-73e5b6fd-54f4c64594-rv7ck"] Jul 15 20:08:06.033543 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:06.033506 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:08:06.238282 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:06.238252 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e155341-bbcd-4dc8-b80a-4c208b9c5374" path="/var/lib/kubelet/pods/4e155341-bbcd-4dc8-b80a-4c208b9c5374/volumes" Jul 15 20:08:06.348888 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:06.348795 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 20:08:06.353834 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:06.353805 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 20:08:06.364609 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:06.364586 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 20:08:06.368978 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:06.368958 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 20:08:08.769780 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.769736 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:08:08.855611 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.855581 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-credential-socket\") pod \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " Jul 15 20:08:08.855822 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.855619 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-workload-certs\") pod \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " Jul 15 20:08:08.855822 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.855657 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-token\") pod \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " Jul 15 20:08:08.855822 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.855690 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-data\") pod \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " Jul 15 20:08:08.855822 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.855723 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istiod-ca-cert\") pod \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " Jul 15 20:08:08.855822 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.855785 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-podinfo\") pod \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " Jul 15 20:08:08.855822 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.855814 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-envoy\") pod \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " Jul 15 20:08:08.856185 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.855856 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brpdm\" (UniqueName: \"kubernetes.io/projected/bfb2c523-a0dc-4e92-aaed-534f1b611dae-kube-api-access-brpdm\") pod \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " Jul 15 20:08:08.856185 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.855880 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-workload-socket\") pod \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\" (UID: \"bfb2c523-a0dc-4e92-aaed-534f1b611dae\") " Jul 15 20:08:08.856185 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.855916 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bfb2c523-a0dc-4e92-aaed-534f1b611dae" (UID: "bfb2c523-a0dc-4e92-aaed-534f1b611dae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:08:08.856185 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.856000 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bfb2c523-a0dc-4e92-aaed-534f1b611dae" (UID: "bfb2c523-a0dc-4e92-aaed-534f1b611dae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:08:08.856408 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.856239 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bfb2c523-a0dc-4e92-aaed-534f1b611dae" (UID: "bfb2c523-a0dc-4e92-aaed-534f1b611dae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:08:08.856408 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.856376 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-credential-socket\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:08:08.856408 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.856396 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-workload-certs\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:08:08.856571 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.856411 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istiod-ca-cert\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:08:08.856635 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.856583 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bfb2c523-a0dc-4e92-aaed-534f1b611dae" (UID: "bfb2c523-a0dc-4e92-aaed-534f1b611dae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:08:08.856782 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.856724 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-data" (OuterVolumeSpecName: "istio-data") pod "bfb2c523-a0dc-4e92-aaed-534f1b611dae" (UID: "bfb2c523-a0dc-4e92-aaed-534f1b611dae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:08:08.858378 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.858333 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-token" (OuterVolumeSpecName: "istio-token") pod "bfb2c523-a0dc-4e92-aaed-534f1b611dae" (UID: "bfb2c523-a0dc-4e92-aaed-534f1b611dae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:08:08.858470 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.858373 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bfb2c523-a0dc-4e92-aaed-534f1b611dae" (UID: "bfb2c523-a0dc-4e92-aaed-534f1b611dae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:08:08.858603 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.858584 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bfb2c523-a0dc-4e92-aaed-534f1b611dae" (UID: "bfb2c523-a0dc-4e92-aaed-534f1b611dae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:08:08.858732 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.858708 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb2c523-a0dc-4e92-aaed-534f1b611dae-kube-api-access-brpdm" (OuterVolumeSpecName: "kube-api-access-brpdm") pod "bfb2c523-a0dc-4e92-aaed-534f1b611dae" (UID: "bfb2c523-a0dc-4e92-aaed-534f1b611dae"). InnerVolumeSpecName "kube-api-access-brpdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:08:08.957352 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.957304 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-token\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:08:08.957352 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.957340 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-data\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:08:08.957352 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.957349 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-podinfo\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:08:08.957352 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.957358 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-istio-envoy\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:08:08.957698 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.957367 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brpdm\" (UniqueName: \"kubernetes.io/projected/bfb2c523-a0dc-4e92-aaed-534f1b611dae-kube-api-access-brpdm\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:08:08.957698 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:08.957377 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb2c523-a0dc-4e92-aaed-534f1b611dae-workload-socket\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:08:09.098135 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:09.098040 2554 generic.go:356] "Generic (PLEG): container finished" podID="bfb2c523-a0dc-4e92-aaed-534f1b611dae" containerID="c5da4df297dba4050861853307a4fc762c96b91fc2a5900303cea5f8aac93cc8" exitCode=0 Jul 15 20:08:09.098135 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:09.098094 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" Jul 15 20:08:09.098370 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:09.098129 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" event={"ID":"bfb2c523-a0dc-4e92-aaed-534f1b611dae","Type":"ContainerDied","Data":"c5da4df297dba4050861853307a4fc762c96b91fc2a5900303cea5f8aac93cc8"} Jul 15 20:08:09.098370 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:09.098192 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2" event={"ID":"bfb2c523-a0dc-4e92-aaed-534f1b611dae","Type":"ContainerDied","Data":"d0d76ad0f7bcd1c30613eabfa032894e16f76c9dae2d611467e42e58cf29e196"} Jul 15 20:08:09.098370 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:09.098223 2554 scope.go:117] "RemoveContainer" containerID="c5da4df297dba4050861853307a4fc762c96b91fc2a5900303cea5f8aac93cc8" Jul 15 20:08:09.107821 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:09.107803 2554 scope.go:117] "RemoveContainer" containerID="45f9b1c1d47812fe45d340b5d58e3f6ccd0602aef2b35ab931fb632c2bebe597" Jul 15 20:08:09.116065 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:09.116046 2554 scope.go:117] "RemoveContainer" containerID="c5da4df297dba4050861853307a4fc762c96b91fc2a5900303cea5f8aac93cc8" Jul 15 20:08:09.116331 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:08:09.116310 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5da4df297dba4050861853307a4fc762c96b91fc2a5900303cea5f8aac93cc8\": container with ID starting with c5da4df297dba4050861853307a4fc762c96b91fc2a5900303cea5f8aac93cc8 not found: ID does not exist" containerID="c5da4df297dba4050861853307a4fc762c96b91fc2a5900303cea5f8aac93cc8" Jul 15 20:08:09.116376 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:09.116340 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5da4df297dba4050861853307a4fc762c96b91fc2a5900303cea5f8aac93cc8"} err="failed to get container status \"c5da4df297dba4050861853307a4fc762c96b91fc2a5900303cea5f8aac93cc8\": rpc error: code = NotFound desc = could not find container \"c5da4df297dba4050861853307a4fc762c96b91fc2a5900303cea5f8aac93cc8\": container with ID starting with c5da4df297dba4050861853307a4fc762c96b91fc2a5900303cea5f8aac93cc8 not found: ID does not exist" Jul 15 20:08:09.116376 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:09.116373 2554 scope.go:117] "RemoveContainer" containerID="45f9b1c1d47812fe45d340b5d58e3f6ccd0602aef2b35ab931fb632c2bebe597" Jul 15 20:08:09.116596 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:08:09.116576 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45f9b1c1d47812fe45d340b5d58e3f6ccd0602aef2b35ab931fb632c2bebe597\": container with ID starting with 45f9b1c1d47812fe45d340b5d58e3f6ccd0602aef2b35ab931fb632c2bebe597 not found: ID does not exist" containerID="45f9b1c1d47812fe45d340b5d58e3f6ccd0602aef2b35ab931fb632c2bebe597" Jul 15 20:08:09.116647 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:09.116604 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45f9b1c1d47812fe45d340b5d58e3f6ccd0602aef2b35ab931fb632c2bebe597"} err="failed to get container status \"45f9b1c1d47812fe45d340b5d58e3f6ccd0602aef2b35ab931fb632c2bebe597\": rpc error: code = NotFound desc = could not find container \"45f9b1c1d47812fe45d340b5d58e3f6ccd0602aef2b35ab931fb632c2bebe597\": container with ID starting with 45f9b1c1d47812fe45d340b5d58e3f6ccd0602aef2b35ab931fb632c2bebe597 not found: ID does not exist" Jul 15 20:08:09.134152 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:09.134128 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2"] Jul 15 20:08:09.142314 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:09.142294 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-73e5b6fd-gw-openshift-default-55c8994b59-9bfx2"] Jul 15 20:08:10.044704 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.044676 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:10.197274 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.197240 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:10.239805 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.239774 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfb2c523-a0dc-4e92-aaed-534f1b611dae" path="/var/lib/kubelet/pods/bfb2c523-a0dc-4e92-aaed-534f1b611dae/volumes" Jul 15 20:08:10.304334 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.304246 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:10.449793 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.449732 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj"] Jul 15 20:08:10.450122 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.450109 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e155341-bbcd-4dc8-b80a-4c208b9c5374" containerName="maas-api" Jul 15 20:08:10.450166 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.450125 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e155341-bbcd-4dc8-b80a-4c208b9c5374" containerName="maas-api" Jul 15 20:08:10.450166 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.450137 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b" containerName="revoke-keys" Jul 15 20:08:10.450166 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.450142 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b" containerName="revoke-keys" Jul 15 20:08:10.450261 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.450164 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb2c523-a0dc-4e92-aaed-534f1b611dae" containerName="istio-proxy" Jul 15 20:08:10.450261 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.450173 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb2c523-a0dc-4e92-aaed-534f1b611dae" containerName="istio-proxy" Jul 15 20:08:10.450261 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.450180 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb2c523-a0dc-4e92-aaed-534f1b611dae" containerName="istio-proxy" Jul 15 20:08:10.450261 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.450186 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb2c523-a0dc-4e92-aaed-534f1b611dae" containerName="istio-proxy" Jul 15 20:08:10.450261 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.450197 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85a9717a-2f91-48a3-b207-4204901a8e69" containerName="revoke-keys" Jul 15 20:08:10.450261 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.450202 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="85a9717a-2f91-48a3-b207-4204901a8e69" containerName="revoke-keys" Jul 15 20:08:10.450261 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.450262 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb2c523-a0dc-4e92-aaed-534f1b611dae" containerName="istio-proxy" Jul 15 20:08:10.450461 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.450270 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb2c523-a0dc-4e92-aaed-534f1b611dae" containerName="istio-proxy" Jul 15 20:08:10.450461 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.450277 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="85a9717a-2f91-48a3-b207-4204901a8e69" containerName="revoke-keys" Jul 15 20:08:10.450461 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.450284 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e155341-bbcd-4dc8-b80a-4c208b9c5374" containerName="maas-api" Jul 15 20:08:10.450461 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.450290 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a64ca88-0ff7-4608-9b0b-bf054bd3ce4b" containerName="revoke-keys" Jul 15 20:08:10.454974 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.454953 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.459173 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.459156 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7a4fe348-openshift-default-dockercfg-ml667\"" Jul 15 20:08:10.470282 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.470258 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-podinfo\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.470369 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.470289 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d859f1-98d9-42e9-a28f-90f08be8d204-istiod-ca-cert\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.470369 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.470315 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-workload-socket\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.470369 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.470340 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kj5k\" (UniqueName: \"kubernetes.io/projected/51d859f1-98d9-42e9-a28f-90f08be8d204-kube-api-access-9kj5k\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.470612 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.470461 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-data\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.470612 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.470495 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-token\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.470612 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.470523 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-credential-socket\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.470612 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.470598 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-workload-certs\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.470790 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.470641 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-envoy\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.478808 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.478784 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj"] Jul 15 20:08:10.478989 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.478972 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:10.571445 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.571360 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-workload-socket\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.571445 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.571400 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kj5k\" (UniqueName: \"kubernetes.io/projected/51d859f1-98d9-42e9-a28f-90f08be8d204-kube-api-access-9kj5k\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.571445 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.571443 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-data\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.571698 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.571468 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-token\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.571698 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.571493 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-credential-socket\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.571698 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.571530 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-workload-certs\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.571698 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.571684 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-envoy\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.571967 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.571732 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-podinfo\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.571967 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.571790 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d859f1-98d9-42e9-a28f-90f08be8d204-istiod-ca-cert\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.571967 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.571860 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-data\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.571967 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.571859 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-workload-socket\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.571967 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.571943 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-workload-certs\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.572225 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.571991 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-credential-socket\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.572461 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.572439 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d859f1-98d9-42e9-a28f-90f08be8d204-istiod-ca-cert\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.573987 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.573970 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-envoy\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.574317 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.574294 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-podinfo\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.586355 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.586326 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-token\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.586799 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.586781 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kj5k\" (UniqueName: \"kubernetes.io/projected/51d859f1-98d9-42e9-a28f-90f08be8d204-kube-api-access-9kj5k\") pod \"e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.764796 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.764764 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:10.922574 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.922542 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj"] Jul 15 20:08:10.923945 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:08:10.923914 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d859f1_98d9_42e9_a28f_90f08be8d204.slice/crio-f891c48f4f628a6024c148c64005337e4a543f1b72f3ad5dc5e19ac6892fe227 WatchSource:0}: Error finding container f891c48f4f628a6024c148c64005337e4a543f1b72f3ad5dc5e19ac6892fe227: Status 404 returned error can't find the container with id f891c48f4f628a6024c148c64005337e4a543f1b72f3ad5dc5e19ac6892fe227 Jul 15 20:08:10.926805 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.926776 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:08:10.927334 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.927291 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:08:10.927439 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:10.927410 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:08:11.113642 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:11.113545 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" event={"ID":"51d859f1-98d9-42e9-a28f-90f08be8d204","Type":"ContainerStarted","Data":"11e514313c84b127be933aa497610695a7ff2cb867a01d83c56c8ef5121affd7"} Jul 15 20:08:11.114068 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:11.113640 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" event={"ID":"51d859f1-98d9-42e9-a28f-90f08be8d204","Type":"ContainerStarted","Data":"f891c48f4f628a6024c148c64005337e4a543f1b72f3ad5dc5e19ac6892fe227"} Jul 15 20:08:11.142743 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:11.142668 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" podStartSLOduration=1.142650688 podStartE2EDuration="1.142650688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:08:11.141025325 +0000 UTC m=+1805.385531785" watchObservedRunningTime="2026-07-15 20:08:11.142650688 +0000 UTC m=+1805.387157150" Jul 15 20:08:11.188910 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:11.188878 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:11.686991 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:11.686961 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:11.766003 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:11.765960 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:11.767592 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:11.767564 2554 patch_prober.go:28] interesting pod/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 15 20:08:11.767799 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:11.767614 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" podUID="51d859f1-98d9-42e9-a28f-90f08be8d204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 15 20:08:11.778192 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:11.778164 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:11.906450 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:11.906415 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf"] Jul 15 20:08:11.910474 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:11.910455 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" Jul 15 20:08:11.913380 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:11.913354 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c5cgj\"" Jul 15 20:08:11.914420 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:11.914308 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7a4fe348-serving-cert\"" Jul 15 20:08:11.914420 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:11.914381 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 20:08:11.914420 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:11.914398 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 20:08:11.922031 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:11.922010 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf"] Jul 15 20:08:11.983852 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:11.983734 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c678fb4b-e102-4c46-bd02-67a2c7446527-maas-api-tls\") pod \"maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf\" (UID: \"c678fb4b-e102-4c46-bd02-67a2c7446527\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" Jul 15 20:08:11.984031 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:11.983882 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c678fb4b-e102-4c46-bd02-67a2c7446527-openshift-service-ca\") pod \"maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf\" (UID: \"c678fb4b-e102-4c46-bd02-67a2c7446527\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" Jul 15 20:08:11.984031 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:11.983922 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvlqv\" (UniqueName: \"kubernetes.io/projected/c678fb4b-e102-4c46-bd02-67a2c7446527-kube-api-access-xvlqv\") pod \"maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf\" (UID: \"c678fb4b-e102-4c46-bd02-67a2c7446527\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" Jul 15 20:08:12.084830 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:12.084785 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvlqv\" (UniqueName: \"kubernetes.io/projected/c678fb4b-e102-4c46-bd02-67a2c7446527-kube-api-access-xvlqv\") pod \"maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf\" (UID: \"c678fb4b-e102-4c46-bd02-67a2c7446527\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" Jul 15 20:08:12.085008 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:12.084870 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c678fb4b-e102-4c46-bd02-67a2c7446527-maas-api-tls\") pod \"maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf\" (UID: \"c678fb4b-e102-4c46-bd02-67a2c7446527\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" Jul 15 20:08:12.085008 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:12.084954 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c678fb4b-e102-4c46-bd02-67a2c7446527-openshift-service-ca\") pod \"maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf\" (UID: \"c678fb4b-e102-4c46-bd02-67a2c7446527\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" Jul 15 20:08:12.085088 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:08:12.085015 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-serving-cert: secret "maas-api-e2e-ait-7a4fe348-serving-cert" not found Jul 15 20:08:12.085088 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:08:12.085083 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c678fb4b-e102-4c46-bd02-67a2c7446527-maas-api-tls podName:c678fb4b-e102-4c46-bd02-67a2c7446527 nodeName:}" failed. No retries permitted until 2026-07-15 20:08:12.585062253 +0000 UTC m=+1806.829568828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c678fb4b-e102-4c46-bd02-67a2c7446527-maas-api-tls") pod "maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" (UID: "c678fb4b-e102-4c46-bd02-67a2c7446527") : secret "maas-api-e2e-ait-7a4fe348-serving-cert" not found Jul 15 20:08:12.085533 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:12.085517 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c678fb4b-e102-4c46-bd02-67a2c7446527-openshift-service-ca\") pod \"maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf\" (UID: \"c678fb4b-e102-4c46-bd02-67a2c7446527\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" Jul 15 20:08:12.096090 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:12.096064 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvlqv\" (UniqueName: \"kubernetes.io/projected/c678fb4b-e102-4c46-bd02-67a2c7446527-kube-api-access-xvlqv\") pod \"maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf\" (UID: \"c678fb4b-e102-4c46-bd02-67a2c7446527\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" Jul 15 20:08:12.589889 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:12.589854 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c678fb4b-e102-4c46-bd02-67a2c7446527-maas-api-tls\") pod \"maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf\" (UID: \"c678fb4b-e102-4c46-bd02-67a2c7446527\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" Jul 15 20:08:12.592502 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:12.592475 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c678fb4b-e102-4c46-bd02-67a2c7446527-maas-api-tls\") pod \"maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf\" (UID: \"c678fb4b-e102-4c46-bd02-67a2c7446527\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" Jul 15 20:08:12.714191 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:12.714156 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:12.769685 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:12.769653 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:12.796236 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:12.796201 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:12.825122 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:12.825086 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" Jul 15 20:08:12.973794 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:12.973765 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf"] Jul 15 20:08:12.975025 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:08:12.974996 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc678fb4b_e102_4c46_bd02_67a2c7446527.slice/crio-dfc8deb4f745e70709819eb86d99ae471d15ee76103162c411f0ee5d4bebe170 WatchSource:0}: Error finding container dfc8deb4f745e70709819eb86d99ae471d15ee76103162c411f0ee5d4bebe170: Status 404 returned error can't find the container with id dfc8deb4f745e70709819eb86d99ae471d15ee76103162c411f0ee5d4bebe170 Jul 15 20:08:13.125134 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:13.125100 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" event={"ID":"c678fb4b-e102-4c46-bd02-67a2c7446527","Type":"ContainerStarted","Data":"dfc8deb4f745e70709819eb86d99ae471d15ee76103162c411f0ee5d4bebe170"} Jul 15 20:08:13.125317 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:13.125292 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:13.126338 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:13.126281 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:13.612947 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:13.612915 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:14.131924 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:14.131887 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" event={"ID":"c678fb4b-e102-4c46-bd02-67a2c7446527","Type":"ContainerStarted","Data":"9c116db4b5287a8ea3dac24db3a0e39b5b6d55651fda9672e1880eba0088d0fe"} Jul 15 20:08:14.132121 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:14.131975 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" Jul 15 20:08:14.154564 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:14.154517 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" podStartSLOduration=2.624547847 podStartE2EDuration="3.154502228s" totalImagesPullingTime="529.954381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:08:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:08:14.152114366 +0000 UTC m=+1808.396620828" watchObservedRunningTime="2026-07-15 20:08:14.154502228 +0000 UTC m=+1808.399008688" Jul 15 20:08:15.641694 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:15.641655 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:08:16.793772 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:16.793713 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89"] Jul 15 20:08:16.797303 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:16.797283 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89" Jul 15 20:08:16.801197 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:16.801159 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tkj8f\"" Jul 15 20:08:16.817187 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:16.817161 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89"] Jul 15 20:08:16.846326 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:16.846274 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5735a97b-063c-40c5-98ca-a03003ad4fc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89\" (UID: \"5735a97b-063c-40c5-98ca-a03003ad4fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89" Jul 15 20:08:16.947467 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:16.947426 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5735a97b-063c-40c5-98ca-a03003ad4fc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89\" (UID: \"5735a97b-063c-40c5-98ca-a03003ad4fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89" Jul 15 20:08:16.948152 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:16.948129 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5735a97b-063c-40c5-98ca-a03003ad4fc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89\" (UID: \"5735a97b-063c-40c5-98ca-a03003ad4fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89" Jul 15 20:08:17.107551 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:17.107452 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89" Jul 15 20:08:17.297780 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:17.297735 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89"] Jul 15 20:08:17.299352 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:08:17.299323 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5735a97b_063c_40c5_98ca_a03003ad4fc3.slice/crio-ec80cdc86204ead685de516f6f89ec65dd951513fd68a67ff98bf91dfd876791 WatchSource:0}: Error finding container ec80cdc86204ead685de516f6f89ec65dd951513fd68a67ff98bf91dfd876791: Status 404 returned error can't find the container with id ec80cdc86204ead685de516f6f89ec65dd951513fd68a67ff98bf91dfd876791 Jul 15 20:08:18.158805 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:18.158767 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89" event={"ID":"5735a97b-063c-40c5-98ca-a03003ad4fc3","Type":"ContainerStarted","Data":"0b9add662f616b6149bdaa8e56b0244bb710f092be998d31dd23836ebb3eccc7"} Jul 15 20:08:18.159257 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:18.158819 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89" event={"ID":"5735a97b-063c-40c5-98ca-a03003ad4fc3","Type":"ContainerStarted","Data":"ec80cdc86204ead685de516f6f89ec65dd951513fd68a67ff98bf91dfd876791"} Jul 15 20:08:18.182425 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:18.182369 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89" podStartSLOduration=2.182353292 podStartE2EDuration="2.182353292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:08:18.180187744 +0000 UTC m=+1812.424694208" watchObservedRunningTime="2026-07-15 20:08:18.182353292 +0000 UTC m=+1812.426859754" Jul 15 20:08:19.165352 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:19.165318 2554 generic.go:356] "Generic (PLEG): container finished" podID="5735a97b-063c-40c5-98ca-a03003ad4fc3" containerID="0b9add662f616b6149bdaa8e56b0244bb710f092be998d31dd23836ebb3eccc7" exitCode=7 Jul 15 20:08:19.165771 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:19.165404 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89" event={"ID":"5735a97b-063c-40c5-98ca-a03003ad4fc3","Type":"ContainerDied","Data":"0b9add662f616b6149bdaa8e56b0244bb710f092be998d31dd23836ebb3eccc7"} Jul 15 20:08:19.165771 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:19.165609 2554 scope.go:117] "RemoveContainer" containerID="0b9add662f616b6149bdaa8e56b0244bb710f092be998d31dd23836ebb3eccc7" Jul 15 20:08:20.149443 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:20.149406 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" Jul 15 20:08:20.172785 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:20.172735 2554 generic.go:356] "Generic (PLEG): container finished" podID="5735a97b-063c-40c5-98ca-a03003ad4fc3" containerID="002f4e87c9b617b99a540e1642d4ef0df17c7131248d10d9ccc92b504aa605e0" exitCode=7 Jul 15 20:08:20.173197 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:20.172826 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89" event={"ID":"5735a97b-063c-40c5-98ca-a03003ad4fc3","Type":"ContainerDied","Data":"002f4e87c9b617b99a540e1642d4ef0df17c7131248d10d9ccc92b504aa605e0"} Jul 15 20:08:20.173197 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:20.172891 2554 scope.go:117] "RemoveContainer" containerID="0b9add662f616b6149bdaa8e56b0244bb710f092be998d31dd23836ebb3eccc7" Jul 15 20:08:20.173197 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:20.173093 2554 scope.go:117] "RemoveContainer" containerID="002f4e87c9b617b99a540e1642d4ef0df17c7131248d10d9ccc92b504aa605e0" Jul 15 20:08:20.173323 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:08:20.173302 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89_odh-ai-gateway-infra(5735a97b-063c-40c5-98ca-a03003ad4fc3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89" podUID="5735a97b-063c-40c5-98ca-a03003ad4fc3" Jul 15 20:08:21.180385 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:21.180363 2554 scope.go:117] "RemoveContainer" containerID="002f4e87c9b617b99a540e1642d4ef0df17c7131248d10d9ccc92b504aa605e0" Jul 15 20:08:21.180772 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:08:21.180535 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89_odh-ai-gateway-infra(5735a97b-063c-40c5-98ca-a03003ad4fc3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89" podUID="5735a97b-063c-40c5-98ca-a03003ad4fc3" Jul 15 20:08:24.198638 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:24.198601 2554 generic.go:356] "Generic (PLEG): container finished" podID="51d859f1-98d9-42e9-a28f-90f08be8d204" containerID="11e514313c84b127be933aa497610695a7ff2cb867a01d83c56c8ef5121affd7" exitCode=0 Jul 15 20:08:24.199069 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:24.198674 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" event={"ID":"51d859f1-98d9-42e9-a28f-90f08be8d204","Type":"ContainerDied","Data":"11e514313c84b127be933aa497610695a7ff2cb867a01d83c56c8ef5121affd7"} Jul 15 20:08:24.199069 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:24.199065 2554 scope.go:117] "RemoveContainer" containerID="11e514313c84b127be933aa497610695a7ff2cb867a01d83c56c8ef5121affd7" Jul 15 20:08:24.200046 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:24.200019 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:08:24.200152 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:24.200084 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:08:24.200152 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:24.200114 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:08:25.206412 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:25.206376 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" event={"ID":"51d859f1-98d9-42e9-a28f-90f08be8d204","Type":"ContainerStarted","Data":"1453ad9f0e41af734797db86eb2573c9f08bf4dad8af530bbeae040001763652"} Jul 15 20:08:25.765795 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:25.765737 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:25.767067 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:25.767040 2554 patch_prober.go:28] interesting pod/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 15 20:08:25.767207 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:25.767098 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" podUID="51d859f1-98d9-42e9-a28f-90f08be8d204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 15 20:08:26.765167 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:26.765129 2554 patch_prober.go:28] interesting pod/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 15 20:08:26.765575 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:26.765186 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" podUID="51d859f1-98d9-42e9-a28f-90f08be8d204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 15 20:08:27.766144 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:27.766111 2554 patch_prober.go:28] interesting pod/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 15 20:08:27.766563 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:27.766170 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" podUID="51d859f1-98d9-42e9-a28f-90f08be8d204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 15 20:08:28.126126 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:28.126005 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:28.567039 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:28.567007 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:28.765623 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:28.765582 2554 patch_prober.go:28] interesting pod/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 15 20:08:28.765847 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:28.765661 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" podUID="51d859f1-98d9-42e9-a28f-90f08be8d204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 15 20:08:29.769539 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:29.769507 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:29.770543 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:29.770522 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:08:35.232919 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:35.232882 2554 scope.go:117] "RemoveContainer" containerID="002f4e87c9b617b99a540e1642d4ef0df17c7131248d10d9ccc92b504aa605e0" Jul 15 20:08:36.276530 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:36.276427 2554 generic.go:356] "Generic (PLEG): container finished" podID="5735a97b-063c-40c5-98ca-a03003ad4fc3" containerID="1dbbff0c81e1f41933e0e6d214f1487d4ed1332b909c6a186cc5a469febaff8f" exitCode=0 Jul 15 20:08:36.276530 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:36.276512 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89" event={"ID":"5735a97b-063c-40c5-98ca-a03003ad4fc3","Type":"ContainerDied","Data":"1dbbff0c81e1f41933e0e6d214f1487d4ed1332b909c6a186cc5a469febaff8f"} Jul 15 20:08:36.277062 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:36.276576 2554 scope.go:117] "RemoveContainer" containerID="002f4e87c9b617b99a540e1642d4ef0df17c7131248d10d9ccc92b504aa605e0" Jul 15 20:08:37.309461 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:37.309419 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89"] Jul 15 20:08:37.428702 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:37.428676 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89" Jul 15 20:08:37.543459 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:37.543369 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5735a97b-063c-40c5-98ca-a03003ad4fc3-maas-api-service-ca\") pod \"5735a97b-063c-40c5-98ca-a03003ad4fc3\" (UID: \"5735a97b-063c-40c5-98ca-a03003ad4fc3\") " Jul 15 20:08:37.543781 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:37.543731 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5735a97b-063c-40c5-98ca-a03003ad4fc3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5735a97b-063c-40c5-98ca-a03003ad4fc3" (UID: "5735a97b-063c-40c5-98ca-a03003ad4fc3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:08:37.644726 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:37.644694 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5735a97b-063c-40c5-98ca-a03003ad4fc3-maas-api-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:08:37.680724 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:37.680688 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:37.781071 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:37.781038 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:38.296391 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:38.296359 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89" Jul 15 20:08:38.296391 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:38.296363 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89" event={"ID":"5735a97b-063c-40c5-98ca-a03003ad4fc3","Type":"ContainerDied","Data":"ec80cdc86204ead685de516f6f89ec65dd951513fd68a67ff98bf91dfd876791"} Jul 15 20:08:38.296637 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:38.296413 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec80cdc86204ead685de516f6f89ec65dd951513fd68a67ff98bf91dfd876791" Jul 15 20:08:38.332490 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:38.332455 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89"] Jul 15 20:08:38.340741 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:38.340712 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-mkl89"] Jul 15 20:08:40.238080 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:40.238047 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5735a97b-063c-40c5-98ca-a03003ad4fc3" path="/var/lib/kubelet/pods/5735a97b-063c-40c5-98ca-a03003ad4fc3/volumes" Jul 15 20:08:47.436415 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:47.436378 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:47.537085 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:08:47.537044 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:16.831186 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:16.831148 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-hbdnj"] Jul 15 20:09:16.831621 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:16.831549 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5735a97b-063c-40c5-98ca-a03003ad4fc3" containerName="revoke-keys" Jul 15 20:09:16.831621 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:16.831569 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5735a97b-063c-40c5-98ca-a03003ad4fc3" containerName="revoke-keys" Jul 15 20:09:16.831621 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:16.831596 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5735a97b-063c-40c5-98ca-a03003ad4fc3" containerName="revoke-keys" Jul 15 20:09:16.831621 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:16.831603 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5735a97b-063c-40c5-98ca-a03003ad4fc3" containerName="revoke-keys" Jul 15 20:09:16.831788 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:16.831662 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5735a97b-063c-40c5-98ca-a03003ad4fc3" containerName="revoke-keys" Jul 15 20:09:16.831788 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:16.831671 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5735a97b-063c-40c5-98ca-a03003ad4fc3" containerName="revoke-keys" Jul 15 20:09:16.834484 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:16.834468 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-hbdnj" Jul 15 20:09:16.837088 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:16.837069 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tkj8f\"" Jul 15 20:09:16.845182 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:16.845157 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-hbdnj"] Jul 15 20:09:16.985683 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:16.985644 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd01ef1-c291-4c45-8537-bc3e6a03759e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a4fe348-hbdnj\" (UID: \"0dd01ef1-c291-4c45-8537-bc3e6a03759e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-hbdnj" Jul 15 20:09:17.087113 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:17.087015 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd01ef1-c291-4c45-8537-bc3e6a03759e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a4fe348-hbdnj\" (UID: \"0dd01ef1-c291-4c45-8537-bc3e6a03759e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-hbdnj" Jul 15 20:09:17.087664 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:17.087642 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd01ef1-c291-4c45-8537-bc3e6a03759e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a4fe348-hbdnj\" (UID: \"0dd01ef1-c291-4c45-8537-bc3e6a03759e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-hbdnj" Jul 15 20:09:17.146337 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:17.146292 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-hbdnj" Jul 15 20:09:17.289908 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:17.289879 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-hbdnj"] Jul 15 20:09:17.290708 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:09:17.290684 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd01ef1_c291_4c45_8537_bc3e6a03759e.slice/crio-8c98d6df98818b8667182f62383bac899f43b959ee45677abb93982c58a20b92 WatchSource:0}: Error finding container 8c98d6df98818b8667182f62383bac899f43b959ee45677abb93982c58a20b92: Status 404 returned error can't find the container with id 8c98d6df98818b8667182f62383bac899f43b959ee45677abb93982c58a20b92 Jul 15 20:09:17.509738 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:17.509703 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-hbdnj" event={"ID":"0dd01ef1-c291-4c45-8537-bc3e6a03759e","Type":"ContainerStarted","Data":"8c98d6df98818b8667182f62383bac899f43b959ee45677abb93982c58a20b92"} Jul 15 20:09:18.516336 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:18.516301 2554 generic.go:356] "Generic (PLEG): container finished" podID="0dd01ef1-c291-4c45-8537-bc3e6a03759e" containerID="65bd3cb9a6d719003933e282934c4f1a7cdd2f558d2dd3ac99bce4a3391bafc1" exitCode=0 Jul 15 20:09:18.516740 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:18.516393 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-hbdnj" event={"ID":"0dd01ef1-c291-4c45-8537-bc3e6a03759e","Type":"ContainerDied","Data":"65bd3cb9a6d719003933e282934c4f1a7cdd2f558d2dd3ac99bce4a3391bafc1"} Jul 15 20:09:19.657919 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:19.657892 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-hbdnj" Jul 15 20:09:19.811678 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:19.811577 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd01ef1-c291-4c45-8537-bc3e6a03759e-maas-api-service-ca\") pod \"0dd01ef1-c291-4c45-8537-bc3e6a03759e\" (UID: \"0dd01ef1-c291-4c45-8537-bc3e6a03759e\") " Jul 15 20:09:19.811952 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:19.811928 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd01ef1-c291-4c45-8537-bc3e6a03759e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0dd01ef1-c291-4c45-8537-bc3e6a03759e" (UID: "0dd01ef1-c291-4c45-8537-bc3e6a03759e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:09:19.912941 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:19.912895 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd01ef1-c291-4c45-8537-bc3e6a03759e-maas-api-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:09:20.529849 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:20.529740 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-hbdnj" event={"ID":"0dd01ef1-c291-4c45-8537-bc3e6a03759e","Type":"ContainerDied","Data":"8c98d6df98818b8667182f62383bac899f43b959ee45677abb93982c58a20b92"} Jul 15 20:09:20.529849 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:20.529844 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c98d6df98818b8667182f62383bac899f43b959ee45677abb93982c58a20b92" Jul 15 20:09:20.530024 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:20.529775 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a4fe348-hbdnj" Jul 15 20:09:27.426532 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:27.426432 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf"] Jul 15 20:09:27.427019 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:27.426703 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" podUID="c678fb4b-e102-4c46-bd02-67a2c7446527" containerName="maas-api" containerID="cri-o://9c116db4b5287a8ea3dac24db3a0e39b5b6d55651fda9672e1880eba0088d0fe" gracePeriod=30 Jul 15 20:09:27.454406 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:27.454377 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:27.571620 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:27.571579 2554 generic.go:356] "Generic (PLEG): container finished" podID="c678fb4b-e102-4c46-bd02-67a2c7446527" containerID="9c116db4b5287a8ea3dac24db3a0e39b5b6d55651fda9672e1880eba0088d0fe" exitCode=0 Jul 15 20:09:27.571815 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:27.571652 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" event={"ID":"c678fb4b-e102-4c46-bd02-67a2c7446527","Type":"ContainerDied","Data":"9c116db4b5287a8ea3dac24db3a0e39b5b6d55651fda9672e1880eba0088d0fe"} Jul 15 20:09:27.689816 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:27.689697 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" Jul 15 20:09:27.785028 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:27.784994 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c678fb4b-e102-4c46-bd02-67a2c7446527-openshift-service-ca\") pod \"c678fb4b-e102-4c46-bd02-67a2c7446527\" (UID: \"c678fb4b-e102-4c46-bd02-67a2c7446527\") " Jul 15 20:09:27.785220 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:27.785040 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c678fb4b-e102-4c46-bd02-67a2c7446527-maas-api-tls\") pod \"c678fb4b-e102-4c46-bd02-67a2c7446527\" (UID: \"c678fb4b-e102-4c46-bd02-67a2c7446527\") " Jul 15 20:09:27.785220 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:27.785101 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvlqv\" (UniqueName: \"kubernetes.io/projected/c678fb4b-e102-4c46-bd02-67a2c7446527-kube-api-access-xvlqv\") pod \"c678fb4b-e102-4c46-bd02-67a2c7446527\" (UID: \"c678fb4b-e102-4c46-bd02-67a2c7446527\") " Jul 15 20:09:27.785376 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:27.785351 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c678fb4b-e102-4c46-bd02-67a2c7446527-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c678fb4b-e102-4c46-bd02-67a2c7446527" (UID: "c678fb4b-e102-4c46-bd02-67a2c7446527"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:09:27.787302 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:27.787268 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c678fb4b-e102-4c46-bd02-67a2c7446527-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c678fb4b-e102-4c46-bd02-67a2c7446527" (UID: "c678fb4b-e102-4c46-bd02-67a2c7446527"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:09:27.787434 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:27.787414 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c678fb4b-e102-4c46-bd02-67a2c7446527-kube-api-access-xvlqv" (OuterVolumeSpecName: "kube-api-access-xvlqv") pod "c678fb4b-e102-4c46-bd02-67a2c7446527" (UID: "c678fb4b-e102-4c46-bd02-67a2c7446527"). InnerVolumeSpecName "kube-api-access-xvlqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:09:27.886526 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:27.886490 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c678fb4b-e102-4c46-bd02-67a2c7446527-maas-api-tls\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:09:27.886526 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:27.886518 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvlqv\" (UniqueName: \"kubernetes.io/projected/c678fb4b-e102-4c46-bd02-67a2c7446527-kube-api-access-xvlqv\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:09:27.886526 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:27.886529 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c678fb4b-e102-4c46-bd02-67a2c7446527-openshift-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:09:28.578772 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:28.578663 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" Jul 15 20:09:28.578772 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:28.578704 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf" event={"ID":"c678fb4b-e102-4c46-bd02-67a2c7446527","Type":"ContainerDied","Data":"dfc8deb4f745e70709819eb86d99ae471d15ee76103162c411f0ee5d4bebe170"} Jul 15 20:09:28.579250 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:28.578778 2554 scope.go:117] "RemoveContainer" containerID="9c116db4b5287a8ea3dac24db3a0e39b5b6d55651fda9672e1880eba0088d0fe" Jul 15 20:09:28.611051 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:28.611018 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf"] Jul 15 20:09:28.631617 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:28.631590 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7a4fe348-78fccc779f-nxrjf"] Jul 15 20:09:30.238639 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:30.238607 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c678fb4b-e102-4c46-bd02-67a2c7446527" path="/var/lib/kubelet/pods/c678fb4b-e102-4c46-bd02-67a2c7446527/volumes" Jul 15 20:09:30.548252 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:30.548168 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:09:31.933464 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:31.933434 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:38.363423 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:38.363389 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:38.460814 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:38.460777 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:42.948768 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:42.948710 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj"] Jul 15 20:09:42.949222 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:42.948924 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" podUID="51d859f1-98d9-42e9-a28f-90f08be8d204" containerName="istio-proxy" containerID="cri-o://1453ad9f0e41af734797db86eb2573c9f08bf4dad8af530bbeae040001763652" gracePeriod=30 Jul 15 20:09:42.990036 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:42.990006 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:43.310868 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:43.310836 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:44.475761 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.475722 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:44.612976 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.612937 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx"] Jul 15 20:09:44.613361 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.613344 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dd01ef1-c291-4c45-8537-bc3e6a03759e" containerName="revoke-keys" Jul 15 20:09:44.613414 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.613364 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dd01ef1-c291-4c45-8537-bc3e6a03759e" containerName="revoke-keys" Jul 15 20:09:44.613414 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.613373 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c678fb4b-e102-4c46-bd02-67a2c7446527" containerName="maas-api" Jul 15 20:09:44.613414 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.613379 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c678fb4b-e102-4c46-bd02-67a2c7446527" containerName="maas-api" Jul 15 20:09:44.613414 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.613393 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5735a97b-063c-40c5-98ca-a03003ad4fc3" containerName="revoke-keys" Jul 15 20:09:44.613414 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.613398 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5735a97b-063c-40c5-98ca-a03003ad4fc3" containerName="revoke-keys" Jul 15 20:09:44.613578 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.613456 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5735a97b-063c-40c5-98ca-a03003ad4fc3" containerName="revoke-keys" Jul 15 20:09:44.613578 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.613466 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c678fb4b-e102-4c46-bd02-67a2c7446527" containerName="maas-api" Jul 15 20:09:44.613578 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.613472 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dd01ef1-c291-4c45-8537-bc3e6a03759e" containerName="revoke-keys" Jul 15 20:09:44.616629 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.616606 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" Jul 15 20:09:44.619125 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.619098 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 20:09:44.620075 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.620051 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-60ba2b89-serving-cert\"" Jul 15 20:09:44.620198 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.620085 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c5cgj\"" Jul 15 20:09:44.620198 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.620148 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 20:09:44.652819 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.652787 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx"] Jul 15 20:09:44.739299 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.739197 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904c9335-33d5-4af7-8c51-8936f0067d40-maas-api-tls\") pod \"maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx\" (UID: \"904c9335-33d5-4af7-8c51-8936f0067d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" Jul 15 20:09:44.739299 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.739237 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904c9335-33d5-4af7-8c51-8936f0067d40-openshift-service-ca\") pod \"maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx\" (UID: \"904c9335-33d5-4af7-8c51-8936f0067d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" Jul 15 20:09:44.739541 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.739374 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8k47\" (UniqueName: \"kubernetes.io/projected/904c9335-33d5-4af7-8c51-8936f0067d40-kube-api-access-z8k47\") pod \"maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx\" (UID: \"904c9335-33d5-4af7-8c51-8936f0067d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" Jul 15 20:09:44.773560 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.773490 2554 patch_prober.go:28] interesting pod/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 20:09:44.774382 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.773981 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" podUID="51d859f1-98d9-42e9-a28f-90f08be8d204" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 20:09:44.840245 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.840208 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904c9335-33d5-4af7-8c51-8936f0067d40-maas-api-tls\") pod \"maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx\" (UID: \"904c9335-33d5-4af7-8c51-8936f0067d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" Jul 15 20:09:44.840245 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.840254 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904c9335-33d5-4af7-8c51-8936f0067d40-openshift-service-ca\") pod \"maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx\" (UID: \"904c9335-33d5-4af7-8c51-8936f0067d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" Jul 15 20:09:44.840521 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.840346 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8k47\" (UniqueName: \"kubernetes.io/projected/904c9335-33d5-4af7-8c51-8936f0067d40-kube-api-access-z8k47\") pod \"maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx\" (UID: \"904c9335-33d5-4af7-8c51-8936f0067d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" Jul 15 20:09:44.840521 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:09:44.840376 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-serving-cert: secret "maas-api-e2e-derive-60ba2b89-serving-cert" not found Jul 15 20:09:44.840521 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:09:44.840473 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/904c9335-33d5-4af7-8c51-8936f0067d40-maas-api-tls podName:904c9335-33d5-4af7-8c51-8936f0067d40 nodeName:}" failed. No retries permitted until 2026-07-15 20:09:45.340451644 +0000 UTC m=+1899.584958106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/904c9335-33d5-4af7-8c51-8936f0067d40-maas-api-tls") pod "maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" (UID: "904c9335-33d5-4af7-8c51-8936f0067d40") : secret "maas-api-e2e-derive-60ba2b89-serving-cert" not found Jul 15 20:09:44.841028 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.841007 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904c9335-33d5-4af7-8c51-8936f0067d40-openshift-service-ca\") pod \"maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx\" (UID: \"904c9335-33d5-4af7-8c51-8936f0067d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" Jul 15 20:09:44.856083 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.856059 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8k47\" (UniqueName: \"kubernetes.io/projected/904c9335-33d5-4af7-8c51-8936f0067d40-kube-api-access-z8k47\") pod \"maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx\" (UID: \"904c9335-33d5-4af7-8c51-8936f0067d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" Jul 15 20:09:44.858133 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:44.858100 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:45.204368 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:45.204339 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:45.344217 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:45.344171 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904c9335-33d5-4af7-8c51-8936f0067d40-maas-api-tls\") pod \"maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx\" (UID: \"904c9335-33d5-4af7-8c51-8936f0067d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" Jul 15 20:09:45.346863 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:45.346844 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904c9335-33d5-4af7-8c51-8936f0067d40-maas-api-tls\") pod \"maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx\" (UID: \"904c9335-33d5-4af7-8c51-8936f0067d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" Jul 15 20:09:45.528347 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:45.528256 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" Jul 15 20:09:45.706229 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:45.706188 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx"] Jul 15 20:09:45.709449 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:09:45.709416 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904c9335_33d5_4af7_8c51_8936f0067d40.slice/crio-97150c53bbf72d9b660341225e41734f425c8d7358211e9d784a0b7b98b2bd7c WatchSource:0}: Error finding container 97150c53bbf72d9b660341225e41734f425c8d7358211e9d784a0b7b98b2bd7c: Status 404 returned error can't find the container with id 97150c53bbf72d9b660341225e41734f425c8d7358211e9d784a0b7b98b2bd7c Jul 15 20:09:45.929115 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:45.929079 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:46.083323 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:46.083288 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:46.678795 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:46.678747 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" event={"ID":"904c9335-33d5-4af7-8c51-8936f0067d40","Type":"ContainerStarted","Data":"97150c53bbf72d9b660341225e41734f425c8d7358211e9d784a0b7b98b2bd7c"} Jul 15 20:09:47.685504 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:47.685461 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" event={"ID":"904c9335-33d5-4af7-8c51-8936f0067d40","Type":"ContainerStarted","Data":"fd548a3d2c69ed33b995e3aab4f62c1422652bf3be574c1e4b050e4a425a13df"} Jul 15 20:09:47.685979 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:47.685593 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" Jul 15 20:09:47.715330 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:47.715269 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" podStartSLOduration=2.750301426 podStartE2EDuration="3.715252014s" totalImagesPullingTime="964.950588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:09:47.714451307 +0000 UTC m=+1901.958957769" watchObservedRunningTime="2026-07-15 20:09:47.715252014 +0000 UTC m=+1901.959758475" Jul 15 20:09:48.206944 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.206914 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:09:48.268957 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.268860 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-data\") pod \"51d859f1-98d9-42e9-a28f-90f08be8d204\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " Jul 15 20:09:48.268957 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.268909 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-workload-certs\") pod \"51d859f1-98d9-42e9-a28f-90f08be8d204\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " Jul 15 20:09:48.268957 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.268931 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-podinfo\") pod \"51d859f1-98d9-42e9-a28f-90f08be8d204\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " Jul 15 20:09:48.268957 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.268951 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-token\") pod \"51d859f1-98d9-42e9-a28f-90f08be8d204\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " Jul 15 20:09:48.269272 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.268983 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d859f1-98d9-42e9-a28f-90f08be8d204-istiod-ca-cert\") pod \"51d859f1-98d9-42e9-a28f-90f08be8d204\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " Jul 15 20:09:48.269272 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.269005 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-credential-socket\") pod \"51d859f1-98d9-42e9-a28f-90f08be8d204\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " Jul 15 20:09:48.269272 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.269075 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-envoy\") pod \"51d859f1-98d9-42e9-a28f-90f08be8d204\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " Jul 15 20:09:48.269272 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.269108 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kj5k\" (UniqueName: \"kubernetes.io/projected/51d859f1-98d9-42e9-a28f-90f08be8d204-kube-api-access-9kj5k\") pod \"51d859f1-98d9-42e9-a28f-90f08be8d204\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " Jul 15 20:09:48.269272 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.269165 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-workload-socket\") pod \"51d859f1-98d9-42e9-a28f-90f08be8d204\" (UID: \"51d859f1-98d9-42e9-a28f-90f08be8d204\") " Jul 15 20:09:48.269272 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.269225 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "51d859f1-98d9-42e9-a28f-90f08be8d204" (UID: "51d859f1-98d9-42e9-a28f-90f08be8d204"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:09:48.269584 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.269442 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "51d859f1-98d9-42e9-a28f-90f08be8d204" (UID: "51d859f1-98d9-42e9-a28f-90f08be8d204"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:09:48.269584 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.269453 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-workload-certs\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:09:48.269796 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.269670 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "51d859f1-98d9-42e9-a28f-90f08be8d204" (UID: "51d859f1-98d9-42e9-a28f-90f08be8d204"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:09:48.269796 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.269736 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d859f1-98d9-42e9-a28f-90f08be8d204-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "51d859f1-98d9-42e9-a28f-90f08be8d204" (UID: "51d859f1-98d9-42e9-a28f-90f08be8d204"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:09:48.270582 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.270544 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-data" (OuterVolumeSpecName: "istio-data") pod "51d859f1-98d9-42e9-a28f-90f08be8d204" (UID: "51d859f1-98d9-42e9-a28f-90f08be8d204"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:09:48.272068 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.272019 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "51d859f1-98d9-42e9-a28f-90f08be8d204" (UID: "51d859f1-98d9-42e9-a28f-90f08be8d204"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:09:48.272068 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.272036 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d859f1-98d9-42e9-a28f-90f08be8d204-kube-api-access-9kj5k" (OuterVolumeSpecName: "kube-api-access-9kj5k") pod "51d859f1-98d9-42e9-a28f-90f08be8d204" (UID: "51d859f1-98d9-42e9-a28f-90f08be8d204"). InnerVolumeSpecName "kube-api-access-9kj5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:09:48.272256 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.272136 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-token" (OuterVolumeSpecName: "istio-token") pod "51d859f1-98d9-42e9-a28f-90f08be8d204" (UID: "51d859f1-98d9-42e9-a28f-90f08be8d204"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:09:48.272334 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.272304 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "51d859f1-98d9-42e9-a28f-90f08be8d204" (UID: "51d859f1-98d9-42e9-a28f-90f08be8d204"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:09:48.369920 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.369888 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-envoy\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:09:48.369920 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.369912 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kj5k\" (UniqueName: \"kubernetes.io/projected/51d859f1-98d9-42e9-a28f-90f08be8d204-kube-api-access-9kj5k\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:09:48.369920 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.369923 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-workload-socket\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:09:48.369920 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.369931 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-data\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:09:48.370186 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.369940 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-podinfo\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:09:48.370186 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.369949 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d859f1-98d9-42e9-a28f-90f08be8d204-istio-token\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:09:48.370186 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.369957 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d859f1-98d9-42e9-a28f-90f08be8d204-istiod-ca-cert\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:09:48.370186 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.369965 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d859f1-98d9-42e9-a28f-90f08be8d204-credential-socket\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:09:48.698991 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.698955 2554 generic.go:356] "Generic (PLEG): container finished" podID="51d859f1-98d9-42e9-a28f-90f08be8d204" containerID="1453ad9f0e41af734797db86eb2573c9f08bf4dad8af530bbeae040001763652" exitCode=0 Jul 15 20:09:48.699457 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.699008 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" Jul 15 20:09:48.699457 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.699047 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" event={"ID":"51d859f1-98d9-42e9-a28f-90f08be8d204","Type":"ContainerDied","Data":"1453ad9f0e41af734797db86eb2573c9f08bf4dad8af530bbeae040001763652"} Jul 15 20:09:48.699457 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.699119 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj" event={"ID":"51d859f1-98d9-42e9-a28f-90f08be8d204","Type":"ContainerDied","Data":"f891c48f4f628a6024c148c64005337e4a543f1b72f3ad5dc5e19ac6892fe227"} Jul 15 20:09:48.699457 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.699143 2554 scope.go:117] "RemoveContainer" containerID="1453ad9f0e41af734797db86eb2573c9f08bf4dad8af530bbeae040001763652" Jul 15 20:09:48.709000 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.708976 2554 scope.go:117] "RemoveContainer" containerID="11e514313c84b127be933aa497610695a7ff2cb867a01d83c56c8ef5121affd7" Jul 15 20:09:48.719592 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.719540 2554 scope.go:117] "RemoveContainer" containerID="1453ad9f0e41af734797db86eb2573c9f08bf4dad8af530bbeae040001763652" Jul 15 20:09:48.720062 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:09:48.720038 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1453ad9f0e41af734797db86eb2573c9f08bf4dad8af530bbeae040001763652\": container with ID starting with 1453ad9f0e41af734797db86eb2573c9f08bf4dad8af530bbeae040001763652 not found: ID does not exist" containerID="1453ad9f0e41af734797db86eb2573c9f08bf4dad8af530bbeae040001763652" Jul 15 20:09:48.720155 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.720071 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1453ad9f0e41af734797db86eb2573c9f08bf4dad8af530bbeae040001763652"} err="failed to get container status \"1453ad9f0e41af734797db86eb2573c9f08bf4dad8af530bbeae040001763652\": rpc error: code = NotFound desc = could not find container \"1453ad9f0e41af734797db86eb2573c9f08bf4dad8af530bbeae040001763652\": container with ID starting with 1453ad9f0e41af734797db86eb2573c9f08bf4dad8af530bbeae040001763652 not found: ID does not exist" Jul 15 20:09:48.720155 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.720090 2554 scope.go:117] "RemoveContainer" containerID="11e514313c84b127be933aa497610695a7ff2cb867a01d83c56c8ef5121affd7" Jul 15 20:09:48.720337 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:09:48.720318 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11e514313c84b127be933aa497610695a7ff2cb867a01d83c56c8ef5121affd7\": container with ID starting with 11e514313c84b127be933aa497610695a7ff2cb867a01d83c56c8ef5121affd7 not found: ID does not exist" containerID="11e514313c84b127be933aa497610695a7ff2cb867a01d83c56c8ef5121affd7" Jul 15 20:09:48.720381 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.720344 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e514313c84b127be933aa497610695a7ff2cb867a01d83c56c8ef5121affd7"} err="failed to get container status \"11e514313c84b127be933aa497610695a7ff2cb867a01d83c56c8ef5121affd7\": rpc error: code = NotFound desc = could not find container \"11e514313c84b127be933aa497610695a7ff2cb867a01d83c56c8ef5121affd7\": container with ID starting with 11e514313c84b127be933aa497610695a7ff2cb867a01d83c56c8ef5121affd7 not found: ID does not exist" Jul 15 20:09:48.726701 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.726675 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj"] Jul 15 20:09:48.728826 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.728804 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7a4fe348-openshift-default-8d86cf699-jnrdj"] Jul 15 20:09:48.831949 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.831919 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk"] Jul 15 20:09:48.832278 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.832265 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51d859f1-98d9-42e9-a28f-90f08be8d204" containerName="istio-proxy" Jul 15 20:09:48.832316 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.832280 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="51d859f1-98d9-42e9-a28f-90f08be8d204" containerName="istio-proxy" Jul 15 20:09:48.832316 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.832298 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51d859f1-98d9-42e9-a28f-90f08be8d204" containerName="istio-proxy" Jul 15 20:09:48.832316 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.832306 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="51d859f1-98d9-42e9-a28f-90f08be8d204" containerName="istio-proxy" Jul 15 20:09:48.832422 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.832373 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="51d859f1-98d9-42e9-a28f-90f08be8d204" containerName="istio-proxy" Jul 15 20:09:48.837112 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.837092 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk" Jul 15 20:09:48.839910 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.839889 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tkj8f\"" Jul 15 20:09:48.853662 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.853638 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk"] Jul 15 20:09:48.874986 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.874958 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8440fd7-d37a-4d91-a3f1-dfdeb6571551-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk\" (UID: \"e8440fd7-d37a-4d91-a3f1-dfdeb6571551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk" Jul 15 20:09:48.976353 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.976259 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8440fd7-d37a-4d91-a3f1-dfdeb6571551-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk\" (UID: \"e8440fd7-d37a-4d91-a3f1-dfdeb6571551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk" Jul 15 20:09:48.976943 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:48.976923 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8440fd7-d37a-4d91-a3f1-dfdeb6571551-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk\" (UID: \"e8440fd7-d37a-4d91-a3f1-dfdeb6571551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk" Jul 15 20:09:49.146991 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:49.146959 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk" Jul 15 20:09:49.284689 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:49.284659 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk"] Jul 15 20:09:49.285586 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:09:49.285559 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8440fd7_d37a_4d91_a3f1_dfdeb6571551.slice/crio-3f99bf3bb2a89488c6074c5154d645e7a8407bd88a6255c5806c73986f1e49a1 WatchSource:0}: Error finding container 3f99bf3bb2a89488c6074c5154d645e7a8407bd88a6255c5806c73986f1e49a1: Status 404 returned error can't find the container with id 3f99bf3bb2a89488c6074c5154d645e7a8407bd88a6255c5806c73986f1e49a1 Jul 15 20:09:49.706645 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:49.706608 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk" event={"ID":"e8440fd7-d37a-4d91-a3f1-dfdeb6571551","Type":"ContainerStarted","Data":"0d9470028b7435fdf87bc54c10cb00ebbcc3eb121e1d13615006e8089c1de1bb"} Jul 15 20:09:49.707107 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:49.706669 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk" event={"ID":"e8440fd7-d37a-4d91-a3f1-dfdeb6571551","Type":"ContainerStarted","Data":"3f99bf3bb2a89488c6074c5154d645e7a8407bd88a6255c5806c73986f1e49a1"} Jul 15 20:09:49.726631 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:49.726578 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk" podStartSLOduration=1.726562406 podStartE2EDuration="1.726562406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:09:49.724704642 +0000 UTC m=+1903.969211104" watchObservedRunningTime="2026-07-15 20:09:49.726562406 +0000 UTC m=+1903.971068867" Jul 15 20:09:50.239106 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:50.239073 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51d859f1-98d9-42e9-a28f-90f08be8d204" path="/var/lib/kubelet/pods/51d859f1-98d9-42e9-a28f-90f08be8d204/volumes" Jul 15 20:09:50.714233 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:50.714197 2554 generic.go:356] "Generic (PLEG): container finished" podID="e8440fd7-d37a-4d91-a3f1-dfdeb6571551" containerID="0d9470028b7435fdf87bc54c10cb00ebbcc3eb121e1d13615006e8089c1de1bb" exitCode=7 Jul 15 20:09:50.714627 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:50.714268 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk" event={"ID":"e8440fd7-d37a-4d91-a3f1-dfdeb6571551","Type":"ContainerDied","Data":"0d9470028b7435fdf87bc54c10cb00ebbcc3eb121e1d13615006e8089c1de1bb"} Jul 15 20:09:50.714627 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:50.714453 2554 scope.go:117] "RemoveContainer" containerID="0d9470028b7435fdf87bc54c10cb00ebbcc3eb121e1d13615006e8089c1de1bb" Jul 15 20:09:51.721495 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:51.721457 2554 generic.go:356] "Generic (PLEG): container finished" podID="e8440fd7-d37a-4d91-a3f1-dfdeb6571551" containerID="3166ce4ee73104b72befa7442397b28dcf407303d8772d43ddc1662fe0391ad2" exitCode=7 Jul 15 20:09:51.721976 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:51.721520 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk" event={"ID":"e8440fd7-d37a-4d91-a3f1-dfdeb6571551","Type":"ContainerDied","Data":"3166ce4ee73104b72befa7442397b28dcf407303d8772d43ddc1662fe0391ad2"} Jul 15 20:09:51.721976 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:51.721598 2554 scope.go:117] "RemoveContainer" containerID="0d9470028b7435fdf87bc54c10cb00ebbcc3eb121e1d13615006e8089c1de1bb" Jul 15 20:09:51.721976 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:51.721822 2554 scope.go:117] "RemoveContainer" containerID="3166ce4ee73104b72befa7442397b28dcf407303d8772d43ddc1662fe0391ad2" Jul 15 20:09:51.722133 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:09:51.722024 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk_odh-ai-gateway-infra(e8440fd7-d37a-4d91-a3f1-dfdeb6571551)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk" podUID="e8440fd7-d37a-4d91-a3f1-dfdeb6571551" Jul 15 20:09:52.728803 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:52.728776 2554 scope.go:117] "RemoveContainer" containerID="3166ce4ee73104b72befa7442397b28dcf407303d8772d43ddc1662fe0391ad2" Jul 15 20:09:52.729276 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:09:52.729040 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk_odh-ai-gateway-infra(e8440fd7-d37a-4d91-a3f1-dfdeb6571551)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk" podUID="e8440fd7-d37a-4d91-a3f1-dfdeb6571551" Jul 15 20:09:53.338176 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:53.338135 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:09:53.705256 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:53.705229 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" Jul 15 20:09:59.544195 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:59.544153 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:59.643127 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:09:59.643078 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:10:03.433945 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:03.433910 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:10:05.232685 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:05.232657 2554 scope.go:117] "RemoveContainer" containerID="3166ce4ee73104b72befa7442397b28dcf407303d8772d43ddc1662fe0391ad2" Jul 15 20:10:05.801853 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:05.801737 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk" event={"ID":"e8440fd7-d37a-4d91-a3f1-dfdeb6571551","Type":"ContainerStarted","Data":"97958d8a76d04d7ffdc28bdeae751a6e1a4210cd6c4413b1c8d98ca25390cb05"} Jul 15 20:10:06.808073 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:06.808039 2554 generic.go:356] "Generic (PLEG): container finished" podID="e8440fd7-d37a-4d91-a3f1-dfdeb6571551" containerID="97958d8a76d04d7ffdc28bdeae751a6e1a4210cd6c4413b1c8d98ca25390cb05" exitCode=0 Jul 15 20:10:06.808490 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:06.808106 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk" event={"ID":"e8440fd7-d37a-4d91-a3f1-dfdeb6571551","Type":"ContainerDied","Data":"97958d8a76d04d7ffdc28bdeae751a6e1a4210cd6c4413b1c8d98ca25390cb05"} Jul 15 20:10:06.808490 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:06.808169 2554 scope.go:117] "RemoveContainer" containerID="3166ce4ee73104b72befa7442397b28dcf407303d8772d43ddc1662fe0391ad2" Jul 15 20:10:06.836767 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:06.836718 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk"] Jul 15 20:10:07.960517 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:07.960489 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk" Jul 15 20:10:08.048538 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:08.048499 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8440fd7-d37a-4d91-a3f1-dfdeb6571551-maas-api-service-ca\") pod \"e8440fd7-d37a-4d91-a3f1-dfdeb6571551\" (UID: \"e8440fd7-d37a-4d91-a3f1-dfdeb6571551\") " Jul 15 20:10:08.048925 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:08.048895 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8440fd7-d37a-4d91-a3f1-dfdeb6571551-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e8440fd7-d37a-4d91-a3f1-dfdeb6571551" (UID: "e8440fd7-d37a-4d91-a3f1-dfdeb6571551"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:10:08.149325 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:08.149232 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8440fd7-d37a-4d91-a3f1-dfdeb6571551-maas-api-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:10:08.821889 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:08.821860 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk" event={"ID":"e8440fd7-d37a-4d91-a3f1-dfdeb6571551","Type":"ContainerDied","Data":"3f99bf3bb2a89488c6074c5154d645e7a8407bd88a6255c5806c73986f1e49a1"} Jul 15 20:10:08.822067 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:08.821907 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f99bf3bb2a89488c6074c5154d645e7a8407bd88a6255c5806c73986f1e49a1" Jul 15 20:10:08.822067 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:08.821867 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk" Jul 15 20:10:08.852479 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:08.852445 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk"] Jul 15 20:10:08.858836 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:08.858804 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-6zzvk"] Jul 15 20:10:10.238105 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:10.238070 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8440fd7-d37a-4d91-a3f1-dfdeb6571551" path="/var/lib/kubelet/pods/e8440fd7-d37a-4d91-a3f1-dfdeb6571551/volumes" Jul 15 20:10:13.040260 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:13.040224 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:10:48.890920 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:48.890885 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-2m9v9"] Jul 15 20:10:48.891400 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:48.891240 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8440fd7-d37a-4d91-a3f1-dfdeb6571551" containerName="revoke-keys" Jul 15 20:10:48.891400 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:48.891251 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8440fd7-d37a-4d91-a3f1-dfdeb6571551" containerName="revoke-keys" Jul 15 20:10:48.891400 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:48.891264 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8440fd7-d37a-4d91-a3f1-dfdeb6571551" containerName="revoke-keys" Jul 15 20:10:48.891400 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:48.891269 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8440fd7-d37a-4d91-a3f1-dfdeb6571551" containerName="revoke-keys" Jul 15 20:10:48.891400 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:48.891344 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8440fd7-d37a-4d91-a3f1-dfdeb6571551" containerName="revoke-keys" Jul 15 20:10:48.891400 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:48.891355 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="51d859f1-98d9-42e9-a28f-90f08be8d204" containerName="istio-proxy" Jul 15 20:10:48.891400 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:48.891366 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8440fd7-d37a-4d91-a3f1-dfdeb6571551" containerName="revoke-keys" Jul 15 20:10:48.891400 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:48.891375 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8440fd7-d37a-4d91-a3f1-dfdeb6571551" containerName="revoke-keys" Jul 15 20:10:48.894269 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:48.894251 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-2m9v9" Jul 15 20:10:48.898895 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:48.898875 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tkj8f\"" Jul 15 20:10:48.907603 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:48.907577 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-2m9v9"] Jul 15 20:10:48.994988 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:48.994948 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a71b1f9c-7312-4760-b9c6-a496525a655a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-60ba2b89-2m9v9\" (UID: \"a71b1f9c-7312-4760-b9c6-a496525a655a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-2m9v9" Jul 15 20:10:49.095554 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:49.095517 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a71b1f9c-7312-4760-b9c6-a496525a655a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-60ba2b89-2m9v9\" (UID: \"a71b1f9c-7312-4760-b9c6-a496525a655a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-2m9v9" Jul 15 20:10:49.096157 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:49.096135 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a71b1f9c-7312-4760-b9c6-a496525a655a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-60ba2b89-2m9v9\" (UID: \"a71b1f9c-7312-4760-b9c6-a496525a655a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-2m9v9" Jul 15 20:10:49.204615 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:49.204581 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-2m9v9" Jul 15 20:10:49.339367 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:49.339340 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-2m9v9"] Jul 15 20:10:49.341613 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:10:49.341583 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71b1f9c_7312_4760_b9c6_a496525a655a.slice/crio-35b0e0536f368f10e0bdff3beff9c1fca0ac4ea36fe3f42006dcd1a38989ebe2 WatchSource:0}: Error finding container 35b0e0536f368f10e0bdff3beff9c1fca0ac4ea36fe3f42006dcd1a38989ebe2: Status 404 returned error can't find the container with id 35b0e0536f368f10e0bdff3beff9c1fca0ac4ea36fe3f42006dcd1a38989ebe2 Jul 15 20:10:50.048424 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:50.048391 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-2m9v9" event={"ID":"a71b1f9c-7312-4760-b9c6-a496525a655a","Type":"ContainerStarted","Data":"e5e81481c189b6add3b1aefdaff6b6e9b0760c7a4b8391a0d0b972bdb9553f6f"} Jul 15 20:10:50.048854 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:50.048444 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-2m9v9" event={"ID":"a71b1f9c-7312-4760-b9c6-a496525a655a","Type":"ContainerStarted","Data":"35b0e0536f368f10e0bdff3beff9c1fca0ac4ea36fe3f42006dcd1a38989ebe2"} Jul 15 20:10:50.068316 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:50.068265 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-2m9v9" podStartSLOduration=2.068248716 podStartE2EDuration="2.068248716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:10:50.066188284 +0000 UTC m=+1964.310694745" watchObservedRunningTime="2026-07-15 20:10:50.068248716 +0000 UTC m=+1964.312755177" Jul 15 20:10:51.055071 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:51.055034 2554 generic.go:356] "Generic (PLEG): container finished" podID="a71b1f9c-7312-4760-b9c6-a496525a655a" containerID="e5e81481c189b6add3b1aefdaff6b6e9b0760c7a4b8391a0d0b972bdb9553f6f" exitCode=0 Jul 15 20:10:51.055458 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:51.055130 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-2m9v9" event={"ID":"a71b1f9c-7312-4760-b9c6-a496525a655a","Type":"ContainerDied","Data":"e5e81481c189b6add3b1aefdaff6b6e9b0760c7a4b8391a0d0b972bdb9553f6f"} Jul 15 20:10:52.193346 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:52.193322 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-2m9v9" Jul 15 20:10:52.323638 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:52.323549 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a71b1f9c-7312-4760-b9c6-a496525a655a-maas-api-service-ca\") pod \"a71b1f9c-7312-4760-b9c6-a496525a655a\" (UID: \"a71b1f9c-7312-4760-b9c6-a496525a655a\") " Jul 15 20:10:52.323964 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:52.323936 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a71b1f9c-7312-4760-b9c6-a496525a655a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a71b1f9c-7312-4760-b9c6-a496525a655a" (UID: "a71b1f9c-7312-4760-b9c6-a496525a655a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:10:52.424964 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:52.424929 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a71b1f9c-7312-4760-b9c6-a496525a655a-maas-api-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:10:53.066570 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:53.066542 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-2m9v9" Jul 15 20:10:53.066744 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:53.066536 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60ba2b89-2m9v9" event={"ID":"a71b1f9c-7312-4760-b9c6-a496525a655a","Type":"ContainerDied","Data":"35b0e0536f368f10e0bdff3beff9c1fca0ac4ea36fe3f42006dcd1a38989ebe2"} Jul 15 20:10:53.066744 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:53.066671 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35b0e0536f368f10e0bdff3beff9c1fca0ac4ea36fe3f42006dcd1a38989ebe2" Jul 15 20:10:59.423684 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:59.423596 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx"] Jul 15 20:10:59.424148 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:59.423887 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" podUID="904c9335-33d5-4af7-8c51-8936f0067d40" containerName="maas-api" containerID="cri-o://fd548a3d2c69ed33b995e3aab4f62c1422652bf3be574c1e4b050e4a425a13df" gracePeriod=30 Jul 15 20:10:59.454196 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:59.454156 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:10:59.675495 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:59.675433 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" Jul 15 20:10:59.787978 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:59.787945 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904c9335-33d5-4af7-8c51-8936f0067d40-maas-api-tls\") pod \"904c9335-33d5-4af7-8c51-8936f0067d40\" (UID: \"904c9335-33d5-4af7-8c51-8936f0067d40\") " Jul 15 20:10:59.788139 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:59.788042 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8k47\" (UniqueName: \"kubernetes.io/projected/904c9335-33d5-4af7-8c51-8936f0067d40-kube-api-access-z8k47\") pod \"904c9335-33d5-4af7-8c51-8936f0067d40\" (UID: \"904c9335-33d5-4af7-8c51-8936f0067d40\") " Jul 15 20:10:59.788139 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:59.788065 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904c9335-33d5-4af7-8c51-8936f0067d40-openshift-service-ca\") pod \"904c9335-33d5-4af7-8c51-8936f0067d40\" (UID: \"904c9335-33d5-4af7-8c51-8936f0067d40\") " Jul 15 20:10:59.788508 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:59.788477 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904c9335-33d5-4af7-8c51-8936f0067d40-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "904c9335-33d5-4af7-8c51-8936f0067d40" (UID: "904c9335-33d5-4af7-8c51-8936f0067d40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:10:59.790257 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:59.790234 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/904c9335-33d5-4af7-8c51-8936f0067d40-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "904c9335-33d5-4af7-8c51-8936f0067d40" (UID: "904c9335-33d5-4af7-8c51-8936f0067d40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:10:59.790338 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:59.790292 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904c9335-33d5-4af7-8c51-8936f0067d40-kube-api-access-z8k47" (OuterVolumeSpecName: "kube-api-access-z8k47") pod "904c9335-33d5-4af7-8c51-8936f0067d40" (UID: "904c9335-33d5-4af7-8c51-8936f0067d40"). InnerVolumeSpecName "kube-api-access-z8k47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:10:59.889382 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:59.889344 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904c9335-33d5-4af7-8c51-8936f0067d40-maas-api-tls\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:10:59.889382 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:59.889379 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8k47\" (UniqueName: \"kubernetes.io/projected/904c9335-33d5-4af7-8c51-8936f0067d40-kube-api-access-z8k47\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:10:59.889382 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:10:59.889391 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904c9335-33d5-4af7-8c51-8936f0067d40-openshift-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:11:00.105486 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:00.105400 2554 generic.go:356] "Generic (PLEG): container finished" podID="904c9335-33d5-4af7-8c51-8936f0067d40" containerID="fd548a3d2c69ed33b995e3aab4f62c1422652bf3be574c1e4b050e4a425a13df" exitCode=0 Jul 15 20:11:00.105486 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:00.105446 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" Jul 15 20:11:00.105667 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:00.105484 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" event={"ID":"904c9335-33d5-4af7-8c51-8936f0067d40","Type":"ContainerDied","Data":"fd548a3d2c69ed33b995e3aab4f62c1422652bf3be574c1e4b050e4a425a13df"} Jul 15 20:11:00.105667 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:00.105542 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx" event={"ID":"904c9335-33d5-4af7-8c51-8936f0067d40","Type":"ContainerDied","Data":"97150c53bbf72d9b660341225e41734f425c8d7358211e9d784a0b7b98b2bd7c"} Jul 15 20:11:00.105667 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:00.105571 2554 scope.go:117] "RemoveContainer" containerID="fd548a3d2c69ed33b995e3aab4f62c1422652bf3be574c1e4b050e4a425a13df" Jul 15 20:11:00.115643 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:00.115624 2554 scope.go:117] "RemoveContainer" containerID="fd548a3d2c69ed33b995e3aab4f62c1422652bf3be574c1e4b050e4a425a13df" Jul 15 20:11:00.115933 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:11:00.115915 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd548a3d2c69ed33b995e3aab4f62c1422652bf3be574c1e4b050e4a425a13df\": container with ID starting with fd548a3d2c69ed33b995e3aab4f62c1422652bf3be574c1e4b050e4a425a13df not found: ID does not exist" containerID="fd548a3d2c69ed33b995e3aab4f62c1422652bf3be574c1e4b050e4a425a13df" Jul 15 20:11:00.115997 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:00.115943 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd548a3d2c69ed33b995e3aab4f62c1422652bf3be574c1e4b050e4a425a13df"} err="failed to get container status \"fd548a3d2c69ed33b995e3aab4f62c1422652bf3be574c1e4b050e4a425a13df\": rpc error: code = NotFound desc = could not find container \"fd548a3d2c69ed33b995e3aab4f62c1422652bf3be574c1e4b050e4a425a13df\": container with ID starting with fd548a3d2c69ed33b995e3aab4f62c1422652bf3be574c1e4b050e4a425a13df not found: ID does not exist" Jul 15 20:11:00.129564 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:00.129531 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx"] Jul 15 20:11:00.138381 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:00.138355 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-60ba2b89-6488c856f8-htdgx"] Jul 15 20:11:00.239145 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:00.239111 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="904c9335-33d5-4af7-8c51-8936f0067d40" path="/var/lib/kubelet/pods/904c9335-33d5-4af7-8c51-8936f0067d40/volumes" Jul 15 20:11:02.448044 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:02.448011 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:11:04.008040 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:04.008000 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:11:10.085329 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:10.085286 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:11:10.195695 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:10.195659 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:11:14.284417 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:14.284382 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:11:14.839305 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:14.839265 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:11:14.941140 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:14.941108 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:11:15.043389 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:15.043354 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:11:16.631143 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:16.631113 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:11:20.830946 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:20.830904 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:11:22.948905 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:22.948868 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:11:23.047912 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:23.047875 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:11:23.573176 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:23.573137 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:11:49.488533 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:49.488495 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:11:55.905236 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:55.905191 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:11:56.098397 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:56.098356 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:11:56.191965 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:11:56.191931 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:06.442273 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:06.442238 2554 scope.go:117] "RemoveContainer" containerID="0da3e00e21a3a6fa8ec33cd7385b6779cf512d182a012462418cab67757b2acb" Jul 15 20:12:12.019907 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:12.019868 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:19.155253 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:19.155218 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:19.272380 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:19.272349 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:19.400573 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:19.400541 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:30.002127 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:30.002047 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:30.934908 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:30.934875 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:37.079090 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:37.079051 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:37.215688 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:37.215650 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:37.367660 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:37.367571 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:37.473927 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:37.473893 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:38.357354 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:38.357322 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:38.461347 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:38.461318 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:38.568428 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:38.568396 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:40.248689 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:40.248659 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:40.742223 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:40.742186 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:12:46.134151 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:46.134115 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:48.115902 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:48.115860 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:48.202626 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:48.202577 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:48.873354 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:48.873315 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:50.645815 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:50.645777 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:12:55.294126 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:55.294071 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:55.409659 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:55.409615 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:55.554807 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:55.554678 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:55.904601 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:55.904519 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww"] Jul 15 20:12:55.904864 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:55.904852 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a71b1f9c-7312-4760-b9c6-a496525a655a" containerName="revoke-keys" Jul 15 20:12:55.904914 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:55.904866 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a71b1f9c-7312-4760-b9c6-a496525a655a" containerName="revoke-keys" Jul 15 20:12:55.904914 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:55.904879 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8440fd7-d37a-4d91-a3f1-dfdeb6571551" containerName="revoke-keys" Jul 15 20:12:55.904914 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:55.904885 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8440fd7-d37a-4d91-a3f1-dfdeb6571551" containerName="revoke-keys" Jul 15 20:12:55.904914 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:55.904901 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="904c9335-33d5-4af7-8c51-8936f0067d40" containerName="maas-api" Jul 15 20:12:55.904914 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:55.904907 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="904c9335-33d5-4af7-8c51-8936f0067d40" containerName="maas-api" Jul 15 20:12:55.905066 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:55.904966 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a71b1f9c-7312-4760-b9c6-a496525a655a" containerName="revoke-keys" Jul 15 20:12:55.905066 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:55.904975 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="904c9335-33d5-4af7-8c51-8936f0067d40" containerName="maas-api" Jul 15 20:12:55.908147 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:55.908131 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:55.923619 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:55.923597 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-82e04801-openshift-default-dockercfg-44qns\"" Jul 15 20:12:55.949398 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:55.949368 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:55.951020 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:55.950996 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww"] Jul 15 20:12:56.081576 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.081534 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg9xb\" (UniqueName: \"kubernetes.io/projected/00f1964a-ad10-4c9b-ad08-7231629227a7-kube-api-access-bg9xb\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.081799 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.081588 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-data\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.081799 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.081667 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-token\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.081799 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.081697 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-workload-certs\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.081799 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.081736 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-credential-socket\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.081982 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.081817 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-podinfo\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.081982 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.081851 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f1964a-ad10-4c9b-ad08-7231629227a7-istiod-ca-cert\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.081982 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.081879 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-envoy\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.081982 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.081916 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-workload-socket\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.182912 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.182878 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg9xb\" (UniqueName: \"kubernetes.io/projected/00f1964a-ad10-4c9b-ad08-7231629227a7-kube-api-access-bg9xb\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.183069 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.182931 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-data\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.183069 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.182971 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-token\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.183069 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.182999 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-workload-certs\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.183069 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.183038 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-credential-socket\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.183260 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.183073 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-podinfo\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.183260 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.183207 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f1964a-ad10-4c9b-ad08-7231629227a7-istiod-ca-cert\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.183364 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.183264 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-envoy\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.183364 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.183315 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-workload-socket\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.183463 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.183422 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-workload-certs\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.183463 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.183447 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-data\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.183571 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.183533 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-credential-socket\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.183702 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.183682 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-workload-socket\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.183937 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.183919 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f1964a-ad10-4c9b-ad08-7231629227a7-istiod-ca-cert\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.185587 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.185569 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-podinfo\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.185650 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.185602 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-envoy\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.193900 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.193877 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-token\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.194042 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.193978 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg9xb\" (UniqueName: \"kubernetes.io/projected/00f1964a-ad10-4c9b-ad08-7231629227a7-kube-api-access-bg9xb\") pod \"e2e-mt-82e04801-openshift-default-74f957555f-cm4ww\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.218934 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.218907 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:56.263185 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.263146 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:56.361663 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.361634 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww"] Jul 15 20:12:56.364122 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:12:56.364086 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f1964a_ad10_4c9b_ad08_7231629227a7.slice/crio-2a9e2bbe5a164fe8153a86fd63f83bd6227bfe7c97cdcfb8171780b5e081ddc7 WatchSource:0}: Error finding container 2a9e2bbe5a164fe8153a86fd63f83bd6227bfe7c97cdcfb8171780b5e081ddc7: Status 404 returned error can't find the container with id 2a9e2bbe5a164fe8153a86fd63f83bd6227bfe7c97cdcfb8171780b5e081ddc7 Jul 15 20:12:56.366366 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.366334 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:12:56.366466 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.366397 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:12:56.366466 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.366426 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:12:56.746004 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.745967 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" event={"ID":"00f1964a-ad10-4c9b-ad08-7231629227a7","Type":"ContainerStarted","Data":"ab3938d8a4e6c68ebef6c9bb2dd47c369914149e11d1ef2236f9ddf9990084f3"} Jul 15 20:12:56.746188 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.746028 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" event={"ID":"00f1964a-ad10-4c9b-ad08-7231629227a7","Type":"ContainerStarted","Data":"2a9e2bbe5a164fe8153a86fd63f83bd6227bfe7c97cdcfb8171780b5e081ddc7"} Jul 15 20:12:56.785899 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:56.785838 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" podStartSLOduration=1.785816027 podStartE2EDuration="1.785816027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:12:56.783525214 +0000 UTC m=+2091.028031666" watchObservedRunningTime="2026-07-15 20:12:56.785816027 +0000 UTC m=+2091.030322490" Jul 15 20:12:57.219836 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:57.219803 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:57.224659 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:57.224631 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:57.751525 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:57.751491 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:57.752669 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:57.752646 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:12:58.950828 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:58.950777 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:59.050226 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.050191 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg"] Jul 15 20:12:59.051514 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.051485 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:59.054065 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.054042 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" Jul 15 20:12:59.057124 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.057104 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c5cgj\"" Jul 15 20:12:59.057834 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.057815 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 20:12:59.057945 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.057930 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-82e04801-serving-cert\"" Jul 15 20:12:59.058003 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.057939 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 20:12:59.071788 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.071741 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg"] Jul 15 20:12:59.144356 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.144314 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:59.210546 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.210450 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43ef9855-0ad4-4a80-97b8-0e32c47a872a-maas-api-tls\") pod \"maas-api-e2e-mt-82e04801-6d485f9fb-vchhg\" (UID: \"43ef9855-0ad4-4a80-97b8-0e32c47a872a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" Jul 15 20:12:59.210546 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.210507 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ef9855-0ad4-4a80-97b8-0e32c47a872a-openshift-service-ca\") pod \"maas-api-e2e-mt-82e04801-6d485f9fb-vchhg\" (UID: \"43ef9855-0ad4-4a80-97b8-0e32c47a872a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" Jul 15 20:12:59.210546 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.210548 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5wcx\" (UniqueName: \"kubernetes.io/projected/43ef9855-0ad4-4a80-97b8-0e32c47a872a-kube-api-access-j5wcx\") pod \"maas-api-e2e-mt-82e04801-6d485f9fb-vchhg\" (UID: \"43ef9855-0ad4-4a80-97b8-0e32c47a872a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" Jul 15 20:12:59.312129 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.312096 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43ef9855-0ad4-4a80-97b8-0e32c47a872a-maas-api-tls\") pod \"maas-api-e2e-mt-82e04801-6d485f9fb-vchhg\" (UID: \"43ef9855-0ad4-4a80-97b8-0e32c47a872a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" Jul 15 20:12:59.312340 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.312148 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ef9855-0ad4-4a80-97b8-0e32c47a872a-openshift-service-ca\") pod \"maas-api-e2e-mt-82e04801-6d485f9fb-vchhg\" (UID: \"43ef9855-0ad4-4a80-97b8-0e32c47a872a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" Jul 15 20:12:59.312340 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.312181 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5wcx\" (UniqueName: \"kubernetes.io/projected/43ef9855-0ad4-4a80-97b8-0e32c47a872a-kube-api-access-j5wcx\") pod \"maas-api-e2e-mt-82e04801-6d485f9fb-vchhg\" (UID: \"43ef9855-0ad4-4a80-97b8-0e32c47a872a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" Jul 15 20:12:59.312911 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.312885 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ef9855-0ad4-4a80-97b8-0e32c47a872a-openshift-service-ca\") pod \"maas-api-e2e-mt-82e04801-6d485f9fb-vchhg\" (UID: \"43ef9855-0ad4-4a80-97b8-0e32c47a872a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" Jul 15 20:12:59.314815 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.314795 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43ef9855-0ad4-4a80-97b8-0e32c47a872a-maas-api-tls\") pod \"maas-api-e2e-mt-82e04801-6d485f9fb-vchhg\" (UID: \"43ef9855-0ad4-4a80-97b8-0e32c47a872a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" Jul 15 20:12:59.320031 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.320003 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:59.322469 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.322440 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5wcx\" (UniqueName: \"kubernetes.io/projected/43ef9855-0ad4-4a80-97b8-0e32c47a872a-kube-api-access-j5wcx\") pod \"maas-api-e2e-mt-82e04801-6d485f9fb-vchhg\" (UID: \"43ef9855-0ad4-4a80-97b8-0e32c47a872a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" Jul 15 20:12:59.365980 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.365945 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" Jul 15 20:12:59.511374 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.511349 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg"] Jul 15 20:12:59.513256 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:12:59.513230 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ef9855_0ad4_4a80_97b8_0e32c47a872a.slice/crio-23e56907bb7febeabbd45319aa7dc14307d37670a77a85af04dc49be6faec9d4 WatchSource:0}: Error finding container 23e56907bb7febeabbd45319aa7dc14307d37670a77a85af04dc49be6faec9d4: Status 404 returned error can't find the container with id 23e56907bb7febeabbd45319aa7dc14307d37670a77a85af04dc49be6faec9d4 Jul 15 20:12:59.514730 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.514712 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 20:12:59.659972 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.659938 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:59.763161 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.763031 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" event={"ID":"43ef9855-0ad4-4a80-97b8-0e32c47a872a","Type":"ContainerStarted","Data":"23e56907bb7febeabbd45319aa7dc14307d37670a77a85af04dc49be6faec9d4"} Jul 15 20:12:59.763330 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:12:59.763223 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:00.769958 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:00.769921 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" event={"ID":"43ef9855-0ad4-4a80-97b8-0e32c47a872a","Type":"ContainerStarted","Data":"9602ef533832b4944975a1ff2097aeeb225dcdaf6c4288810c39bb942beae396"} Jul 15 20:13:00.770388 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:00.770049 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" Jul 15 20:13:00.813306 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:00.813255 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" podStartSLOduration=1.305530685 podStartE2EDuration="1.813238849s" totalImagesPullingTime="507.708164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:12:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:13:00.81088686 +0000 UTC m=+2095.055393320" watchObservedRunningTime="2026-07-15 20:13:00.813238849 +0000 UTC m=+2095.057745315" Jul 15 20:13:03.382628 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:03.382592 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:04.123279 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:04.123229 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:04.247021 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:04.246984 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:04.511101 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:04.511069 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:06.151702 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:06.151618 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:13:06.391771 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:06.391729 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 20:13:06.396367 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:06.396343 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 20:13:06.403231 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:06.403175 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 20:13:06.407600 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:06.407581 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 20:13:06.780630 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:06.780595 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" Jul 15 20:13:10.847738 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:10.847699 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:10.949049 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:10.948991 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:11.227776 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:11.227716 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82e04801-rmlnh"] Jul 15 20:13:11.232578 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:11.232551 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82e04801-rmlnh" Jul 15 20:13:11.236142 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:11.236113 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tkj8f\"" Jul 15 20:13:11.248213 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:11.248183 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82e04801-rmlnh"] Jul 15 20:13:11.315848 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:11.315802 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d011e8df-2238-4396-b9fe-057a915f2254-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-82e04801-rmlnh\" (UID: \"d011e8df-2238-4396-b9fe-057a915f2254\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82e04801-rmlnh" Jul 15 20:13:11.417135 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:11.417081 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d011e8df-2238-4396-b9fe-057a915f2254-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-82e04801-rmlnh\" (UID: \"d011e8df-2238-4396-b9fe-057a915f2254\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82e04801-rmlnh" Jul 15 20:13:11.417724 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:11.417703 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d011e8df-2238-4396-b9fe-057a915f2254-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-82e04801-rmlnh\" (UID: \"d011e8df-2238-4396-b9fe-057a915f2254\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82e04801-rmlnh" Jul 15 20:13:11.547147 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:11.547050 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82e04801-rmlnh" Jul 15 20:13:11.913505 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:11.913474 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82e04801-rmlnh"] Jul 15 20:13:11.914929 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:13:11.914892 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd011e8df_2238_4396_b9fe_057a915f2254.slice/crio-d2aa7622e3d6761d0be5ff3ce4a3ac7dffd57c8e027d52599f45caec76691244 WatchSource:0}: Error finding container d2aa7622e3d6761d0be5ff3ce4a3ac7dffd57c8e027d52599f45caec76691244: Status 404 returned error can't find the container with id d2aa7622e3d6761d0be5ff3ce4a3ac7dffd57c8e027d52599f45caec76691244 Jul 15 20:13:12.839961 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:12.839916 2554 generic.go:356] "Generic (PLEG): container finished" podID="d011e8df-2238-4396-b9fe-057a915f2254" containerID="811e8021f2d78fbb13ed3c1886f3369546a352672a8d11a928fc8e259a827421" exitCode=0 Jul 15 20:13:12.840138 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:12.839960 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82e04801-rmlnh" event={"ID":"d011e8df-2238-4396-b9fe-057a915f2254","Type":"ContainerDied","Data":"811e8021f2d78fbb13ed3c1886f3369546a352672a8d11a928fc8e259a827421"} Jul 15 20:13:12.840138 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:12.840027 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82e04801-rmlnh" event={"ID":"d011e8df-2238-4396-b9fe-057a915f2254","Type":"ContainerStarted","Data":"d2aa7622e3d6761d0be5ff3ce4a3ac7dffd57c8e027d52599f45caec76691244"} Jul 15 20:13:13.793007 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:13.792959 2554 patch_prober.go:28] interesting pod/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 20:13:13.793441 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:13.793041 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" podUID="00f1964a-ad10-4c9b-ad08-7231629227a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 20:13:13.992557 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:13.992528 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82e04801-rmlnh" Jul 15 20:13:14.048353 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:14.048271 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d011e8df-2238-4396-b9fe-057a915f2254-maas-api-service-ca\") pod \"d011e8df-2238-4396-b9fe-057a915f2254\" (UID: \"d011e8df-2238-4396-b9fe-057a915f2254\") " Jul 15 20:13:14.048744 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:14.048712 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d011e8df-2238-4396-b9fe-057a915f2254-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d011e8df-2238-4396-b9fe-057a915f2254" (UID: "d011e8df-2238-4396-b9fe-057a915f2254"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:13:14.149065 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:14.149026 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d011e8df-2238-4396-b9fe-057a915f2254-maas-api-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:13:14.854160 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:14.854075 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82e04801-rmlnh" Jul 15 20:13:14.854160 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:14.854100 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82e04801-rmlnh" event={"ID":"d011e8df-2238-4396-b9fe-057a915f2254","Type":"ContainerDied","Data":"d2aa7622e3d6761d0be5ff3ce4a3ac7dffd57c8e027d52599f45caec76691244"} Jul 15 20:13:14.854160 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:14.854152 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2aa7622e3d6761d0be5ff3ce4a3ac7dffd57c8e027d52599f45caec76691244" Jul 15 20:13:14.855999 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:14.855968 2554 generic.go:356] "Generic (PLEG): container finished" podID="00f1964a-ad10-4c9b-ad08-7231629227a7" containerID="ab3938d8a4e6c68ebef6c9bb2dd47c369914149e11d1ef2236f9ddf9990084f3" exitCode=0 Jul 15 20:13:14.856144 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:14.856020 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" event={"ID":"00f1964a-ad10-4c9b-ad08-7231629227a7","Type":"ContainerDied","Data":"ab3938d8a4e6c68ebef6c9bb2dd47c369914149e11d1ef2236f9ddf9990084f3"} Jul 15 20:13:14.856430 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:14.856401 2554 scope.go:117] "RemoveContainer" containerID="ab3938d8a4e6c68ebef6c9bb2dd47c369914149e11d1ef2236f9ddf9990084f3" Jul 15 20:13:14.857412 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:14.857385 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:13:14.857469 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:14.857456 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:13:14.857514 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:14.857491 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:13:15.863038 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:15.863002 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" event={"ID":"00f1964a-ad10-4c9b-ad08-7231629227a7","Type":"ContainerStarted","Data":"4e2b94215275cfca512eab45f274303e35ebcba65cbc0e37ee124b61c187b509"} Jul 15 20:13:16.219719 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:16.219667 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:13:16.220266 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:16.220230 2554 patch_prober.go:28] interesting pod/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 15 20:13:16.220379 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:16.220308 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" podUID="00f1964a-ad10-4c9b-ad08-7231629227a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 15 20:13:17.219850 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:17.219811 2554 patch_prober.go:28] interesting pod/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 15 20:13:17.220430 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:17.220395 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" podUID="00f1964a-ad10-4c9b-ad08-7231629227a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 15 20:13:18.220237 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:18.220193 2554 patch_prober.go:28] interesting pod/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 15 20:13:18.220887 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:18.220847 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" podUID="00f1964a-ad10-4c9b-ad08-7231629227a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 15 20:13:19.220204 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:19.220147 2554 patch_prober.go:28] interesting pod/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 15 20:13:19.220425 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:19.220241 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" podUID="00f1964a-ad10-4c9b-ad08-7231629227a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 15 20:13:20.220196 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:20.220155 2554 patch_prober.go:28] interesting pod/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 15 20:13:20.220523 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:20.220481 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" podUID="00f1964a-ad10-4c9b-ad08-7231629227a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 15 20:13:21.223390 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:21.223360 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:13:21.223851 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:21.223708 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:13:21.224479 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:21.224461 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:13:21.798438 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:21.798402 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg"] Jul 15 20:13:21.798664 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:21.798627 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" podUID="43ef9855-0ad4-4a80-97b8-0e32c47a872a" containerName="maas-api" containerID="cri-o://9602ef533832b4944975a1ff2097aeeb225dcdaf6c4288810c39bb942beae396" gracePeriod=30 Jul 15 20:13:21.839438 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:21.839399 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:22.062014 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:22.061937 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" Jul 15 20:13:22.124165 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:22.124133 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5wcx\" (UniqueName: \"kubernetes.io/projected/43ef9855-0ad4-4a80-97b8-0e32c47a872a-kube-api-access-j5wcx\") pod \"43ef9855-0ad4-4a80-97b8-0e32c47a872a\" (UID: \"43ef9855-0ad4-4a80-97b8-0e32c47a872a\") " Jul 15 20:13:22.124332 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:22.124211 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43ef9855-0ad4-4a80-97b8-0e32c47a872a-maas-api-tls\") pod \"43ef9855-0ad4-4a80-97b8-0e32c47a872a\" (UID: \"43ef9855-0ad4-4a80-97b8-0e32c47a872a\") " Jul 15 20:13:22.124332 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:22.124232 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ef9855-0ad4-4a80-97b8-0e32c47a872a-openshift-service-ca\") pod \"43ef9855-0ad4-4a80-97b8-0e32c47a872a\" (UID: \"43ef9855-0ad4-4a80-97b8-0e32c47a872a\") " Jul 15 20:13:22.124617 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:22.124593 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ef9855-0ad4-4a80-97b8-0e32c47a872a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "43ef9855-0ad4-4a80-97b8-0e32c47a872a" (UID: "43ef9855-0ad4-4a80-97b8-0e32c47a872a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:13:22.126542 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:22.126522 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ef9855-0ad4-4a80-97b8-0e32c47a872a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "43ef9855-0ad4-4a80-97b8-0e32c47a872a" (UID: "43ef9855-0ad4-4a80-97b8-0e32c47a872a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:13:22.126640 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:22.126541 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ef9855-0ad4-4a80-97b8-0e32c47a872a-kube-api-access-j5wcx" (OuterVolumeSpecName: "kube-api-access-j5wcx") pod "43ef9855-0ad4-4a80-97b8-0e32c47a872a" (UID: "43ef9855-0ad4-4a80-97b8-0e32c47a872a"). InnerVolumeSpecName "kube-api-access-j5wcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:13:22.225003 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:22.224965 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43ef9855-0ad4-4a80-97b8-0e32c47a872a-maas-api-tls\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:13:22.225003 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:22.224998 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ef9855-0ad4-4a80-97b8-0e32c47a872a-openshift-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:13:22.225003 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:22.225007 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5wcx\" (UniqueName: \"kubernetes.io/projected/43ef9855-0ad4-4a80-97b8-0e32c47a872a-kube-api-access-j5wcx\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:13:22.902008 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:22.901974 2554 generic.go:356] "Generic (PLEG): container finished" podID="43ef9855-0ad4-4a80-97b8-0e32c47a872a" containerID="9602ef533832b4944975a1ff2097aeeb225dcdaf6c4288810c39bb942beae396" exitCode=0 Jul 15 20:13:22.902185 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:22.902027 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" Jul 15 20:13:22.902185 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:22.902063 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" event={"ID":"43ef9855-0ad4-4a80-97b8-0e32c47a872a","Type":"ContainerDied","Data":"9602ef533832b4944975a1ff2097aeeb225dcdaf6c4288810c39bb942beae396"} Jul 15 20:13:22.902185 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:22.902117 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg" event={"ID":"43ef9855-0ad4-4a80-97b8-0e32c47a872a","Type":"ContainerDied","Data":"23e56907bb7febeabbd45319aa7dc14307d37670a77a85af04dc49be6faec9d4"} Jul 15 20:13:22.902185 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:22.902141 2554 scope.go:117] "RemoveContainer" containerID="9602ef533832b4944975a1ff2097aeeb225dcdaf6c4288810c39bb942beae396" Jul 15 20:13:22.911365 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:22.911341 2554 scope.go:117] "RemoveContainer" containerID="9602ef533832b4944975a1ff2097aeeb225dcdaf6c4288810c39bb942beae396" Jul 15 20:13:22.911633 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:13:22.911614 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9602ef533832b4944975a1ff2097aeeb225dcdaf6c4288810c39bb942beae396\": container with ID starting with 9602ef533832b4944975a1ff2097aeeb225dcdaf6c4288810c39bb942beae396 not found: ID does not exist" containerID="9602ef533832b4944975a1ff2097aeeb225dcdaf6c4288810c39bb942beae396" Jul 15 20:13:22.911705 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:22.911647 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9602ef533832b4944975a1ff2097aeeb225dcdaf6c4288810c39bb942beae396"} err="failed to get container status \"9602ef533832b4944975a1ff2097aeeb225dcdaf6c4288810c39bb942beae396\": rpc error: code = NotFound desc = could not find container \"9602ef533832b4944975a1ff2097aeeb225dcdaf6c4288810c39bb942beae396\": container with ID starting with 9602ef533832b4944975a1ff2097aeeb225dcdaf6c4288810c39bb942beae396 not found: ID does not exist" Jul 15 20:13:22.920880 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:22.920853 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg"] Jul 15 20:13:22.927841 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:22.927815 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82e04801-6d485f9fb-vchhg"] Jul 15 20:13:24.238564 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:24.238519 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43ef9855-0ad4-4a80-97b8-0e32c47a872a" path="/var/lib/kubelet/pods/43ef9855-0ad4-4a80-97b8-0e32c47a872a/volumes" Jul 15 20:13:24.432504 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:24.432468 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:13:26.333519 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:26.333487 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:32.344149 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:32.344113 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:32.411042 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:32.411011 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:36.969377 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:36.969344 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww"] Jul 15 20:13:36.969766 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:36.969543 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" podUID="00f1964a-ad10-4c9b-ad08-7231629227a7" containerName="istio-proxy" containerID="cri-o://4e2b94215275cfca512eab45f274303e35ebcba65cbc0e37ee124b61c187b509" gracePeriod=30 Jul 15 20:13:37.005884 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:37.005844 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:37.941893 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:37.941850 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:38.059705 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:38.059669 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:38.601206 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:38.601169 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:38.704159 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:38.704124 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:42.219887 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.219863 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:13:42.400854 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.400725 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-podinfo\") pod \"00f1964a-ad10-4c9b-ad08-7231629227a7\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " Jul 15 20:13:42.400854 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.400802 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-workload-socket\") pod \"00f1964a-ad10-4c9b-ad08-7231629227a7\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " Jul 15 20:13:42.400854 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.400830 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bg9xb\" (UniqueName: \"kubernetes.io/projected/00f1964a-ad10-4c9b-ad08-7231629227a7-kube-api-access-bg9xb\") pod \"00f1964a-ad10-4c9b-ad08-7231629227a7\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " Jul 15 20:13:42.401093 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.400940 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-token\") pod \"00f1964a-ad10-4c9b-ad08-7231629227a7\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " Jul 15 20:13:42.401093 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.400971 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-envoy\") pod \"00f1964a-ad10-4c9b-ad08-7231629227a7\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " Jul 15 20:13:42.401093 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.401001 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-data\") pod \"00f1964a-ad10-4c9b-ad08-7231629227a7\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " Jul 15 20:13:42.401253 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.401097 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-workload-certs\") pod \"00f1964a-ad10-4c9b-ad08-7231629227a7\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " Jul 15 20:13:42.401253 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.401155 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-credential-socket\") pod \"00f1964a-ad10-4c9b-ad08-7231629227a7\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " Jul 15 20:13:42.401253 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.401137 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "00f1964a-ad10-4c9b-ad08-7231629227a7" (UID: "00f1964a-ad10-4c9b-ad08-7231629227a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:13:42.401253 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.401215 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f1964a-ad10-4c9b-ad08-7231629227a7-istiod-ca-cert\") pod \"00f1964a-ad10-4c9b-ad08-7231629227a7\" (UID: \"00f1964a-ad10-4c9b-ad08-7231629227a7\") " Jul 15 20:13:42.401459 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.401288 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "00f1964a-ad10-4c9b-ad08-7231629227a7" (UID: "00f1964a-ad10-4c9b-ad08-7231629227a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:13:42.401459 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.401397 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "00f1964a-ad10-4c9b-ad08-7231629227a7" (UID: "00f1964a-ad10-4c9b-ad08-7231629227a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:13:42.401570 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.401535 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-workload-socket\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:13:42.401570 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.401557 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-workload-certs\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:13:42.401675 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.401571 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-credential-socket\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:13:42.401675 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.401615 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f1964a-ad10-4c9b-ad08-7231629227a7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "00f1964a-ad10-4c9b-ad08-7231629227a7" (UID: "00f1964a-ad10-4c9b-ad08-7231629227a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:13:42.402163 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.402138 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-data" (OuterVolumeSpecName: "istio-data") pod "00f1964a-ad10-4c9b-ad08-7231629227a7" (UID: "00f1964a-ad10-4c9b-ad08-7231629227a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:13:42.403423 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.403399 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-token" (OuterVolumeSpecName: "istio-token") pod "00f1964a-ad10-4c9b-ad08-7231629227a7" (UID: "00f1964a-ad10-4c9b-ad08-7231629227a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:13:42.403423 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.403408 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "00f1964a-ad10-4c9b-ad08-7231629227a7" (UID: "00f1964a-ad10-4c9b-ad08-7231629227a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:13:42.403701 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.403680 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "00f1964a-ad10-4c9b-ad08-7231629227a7" (UID: "00f1964a-ad10-4c9b-ad08-7231629227a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:13:42.403794 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.403697 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f1964a-ad10-4c9b-ad08-7231629227a7-kube-api-access-bg9xb" (OuterVolumeSpecName: "kube-api-access-bg9xb") pod "00f1964a-ad10-4c9b-ad08-7231629227a7" (UID: "00f1964a-ad10-4c9b-ad08-7231629227a7"). InnerVolumeSpecName "kube-api-access-bg9xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:13:42.502606 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.502566 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f1964a-ad10-4c9b-ad08-7231629227a7-istiod-ca-cert\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:13:42.502606 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.502597 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-podinfo\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:13:42.502606 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.502607 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg9xb\" (UniqueName: \"kubernetes.io/projected/00f1964a-ad10-4c9b-ad08-7231629227a7-kube-api-access-bg9xb\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:13:42.502606 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.502615 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-token\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:13:42.502940 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.502623 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-envoy\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:13:42.502940 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:42.502631 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f1964a-ad10-4c9b-ad08-7231629227a7-istio-data\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:13:43.027996 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:43.027956 2554 generic.go:356] "Generic (PLEG): container finished" podID="00f1964a-ad10-4c9b-ad08-7231629227a7" containerID="4e2b94215275cfca512eab45f274303e35ebcba65cbc0e37ee124b61c187b509" exitCode=0 Jul 15 20:13:43.028178 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:43.028021 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" Jul 15 20:13:43.028178 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:43.028050 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" event={"ID":"00f1964a-ad10-4c9b-ad08-7231629227a7","Type":"ContainerDied","Data":"4e2b94215275cfca512eab45f274303e35ebcba65cbc0e37ee124b61c187b509"} Jul 15 20:13:43.028178 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:43.028126 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww" event={"ID":"00f1964a-ad10-4c9b-ad08-7231629227a7","Type":"ContainerDied","Data":"2a9e2bbe5a164fe8153a86fd63f83bd6227bfe7c97cdcfb8171780b5e081ddc7"} Jul 15 20:13:43.028178 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:43.028149 2554 scope.go:117] "RemoveContainer" containerID="4e2b94215275cfca512eab45f274303e35ebcba65cbc0e37ee124b61c187b509" Jul 15 20:13:43.038448 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:43.038429 2554 scope.go:117] "RemoveContainer" containerID="ab3938d8a4e6c68ebef6c9bb2dd47c369914149e11d1ef2236f9ddf9990084f3" Jul 15 20:13:43.048855 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:43.047148 2554 scope.go:117] "RemoveContainer" containerID="4e2b94215275cfca512eab45f274303e35ebcba65cbc0e37ee124b61c187b509" Jul 15 20:13:43.048855 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:13:43.047453 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2b94215275cfca512eab45f274303e35ebcba65cbc0e37ee124b61c187b509\": container with ID starting with 4e2b94215275cfca512eab45f274303e35ebcba65cbc0e37ee124b61c187b509 not found: ID does not exist" containerID="4e2b94215275cfca512eab45f274303e35ebcba65cbc0e37ee124b61c187b509" Jul 15 20:13:43.048855 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:43.047484 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2b94215275cfca512eab45f274303e35ebcba65cbc0e37ee124b61c187b509"} err="failed to get container status \"4e2b94215275cfca512eab45f274303e35ebcba65cbc0e37ee124b61c187b509\": rpc error: code = NotFound desc = could not find container \"4e2b94215275cfca512eab45f274303e35ebcba65cbc0e37ee124b61c187b509\": container with ID starting with 4e2b94215275cfca512eab45f274303e35ebcba65cbc0e37ee124b61c187b509 not found: ID does not exist" Jul 15 20:13:43.048855 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:43.047505 2554 scope.go:117] "RemoveContainer" containerID="ab3938d8a4e6c68ebef6c9bb2dd47c369914149e11d1ef2236f9ddf9990084f3" Jul 15 20:13:43.048855 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:13:43.047774 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab3938d8a4e6c68ebef6c9bb2dd47c369914149e11d1ef2236f9ddf9990084f3\": container with ID starting with ab3938d8a4e6c68ebef6c9bb2dd47c369914149e11d1ef2236f9ddf9990084f3 not found: ID does not exist" containerID="ab3938d8a4e6c68ebef6c9bb2dd47c369914149e11d1ef2236f9ddf9990084f3" Jul 15 20:13:43.048855 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:43.047799 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab3938d8a4e6c68ebef6c9bb2dd47c369914149e11d1ef2236f9ddf9990084f3"} err="failed to get container status \"ab3938d8a4e6c68ebef6c9bb2dd47c369914149e11d1ef2236f9ddf9990084f3\": rpc error: code = NotFound desc = could not find container \"ab3938d8a4e6c68ebef6c9bb2dd47c369914149e11d1ef2236f9ddf9990084f3\": container with ID starting with ab3938d8a4e6c68ebef6c9bb2dd47c369914149e11d1ef2236f9ddf9990084f3 not found: ID does not exist" Jul 15 20:13:43.054799 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:43.054778 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww"] Jul 15 20:13:43.058559 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:43.058540 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-82e04801-openshift-default-74f957555f-cm4ww"] Jul 15 20:13:43.644549 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:43.644513 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:43.771472 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:43.771440 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:43.875593 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:43.875553 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:44.040801 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:44.040743 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:44.238565 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:44.238530 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00f1964a-ad10-4c9b-ad08-7231629227a7" path="/var/lib/kubelet/pods/00f1964a-ad10-4c9b-ad08-7231629227a7/volumes" Jul 15 20:13:49.941131 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:49.941091 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:50.049290 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:50.049253 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:56.730829 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:56.730745 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:57.034475 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.034377 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97"] Jul 15 20:13:57.034904 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.034886 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00f1964a-ad10-4c9b-ad08-7231629227a7" containerName="istio-proxy" Jul 15 20:13:57.035002 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.034907 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="00f1964a-ad10-4c9b-ad08-7231629227a7" containerName="istio-proxy" Jul 15 20:13:57.035002 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.034923 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00f1964a-ad10-4c9b-ad08-7231629227a7" containerName="istio-proxy" Jul 15 20:13:57.035002 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.034931 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="00f1964a-ad10-4c9b-ad08-7231629227a7" containerName="istio-proxy" Jul 15 20:13:57.035002 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.034964 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d011e8df-2238-4396-b9fe-057a915f2254" containerName="revoke-keys" Jul 15 20:13:57.035002 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.034973 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d011e8df-2238-4396-b9fe-057a915f2254" containerName="revoke-keys" Jul 15 20:13:57.035002 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.034990 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43ef9855-0ad4-4a80-97b8-0e32c47a872a" containerName="maas-api" Jul 15 20:13:57.035002 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.034998 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="43ef9855-0ad4-4a80-97b8-0e32c47a872a" containerName="maas-api" Jul 15 20:13:57.035368 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.035076 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="43ef9855-0ad4-4a80-97b8-0e32c47a872a" containerName="maas-api" Jul 15 20:13:57.035368 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.035085 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="00f1964a-ad10-4c9b-ad08-7231629227a7" containerName="istio-proxy" Jul 15 20:13:57.035368 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.035093 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d011e8df-2238-4396-b9fe-057a915f2254" containerName="revoke-keys" Jul 15 20:13:57.038658 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.038607 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.041460 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.041441 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8950fff5-openshift-default-dockercfg-ttlb6\"" Jul 15 20:13:57.063938 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.063907 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97"] Jul 15 20:13:57.080312 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.080271 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:57.112063 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.112030 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-workload-certs\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.112262 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.112105 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-workload-socket\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.112262 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.112140 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-data\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.112262 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.112166 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-envoy\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.112262 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.112205 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-token\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.112262 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.112256 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-credential-socket\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.112495 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.112282 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sslw\" (UniqueName: \"kubernetes.io/projected/a5f9b52d-c597-442c-bdc4-28566eed7ba6-kube-api-access-6sslw\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.112495 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.112333 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-podinfo\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.112495 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.112357 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istiod-ca-cert\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.169050 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.169008 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:57.213028 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.212924 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-podinfo\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.213201 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.213050 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istiod-ca-cert\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.213201 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.213081 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-workload-certs\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.213201 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.213132 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-workload-socket\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.213201 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.213156 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-data\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.213201 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.213173 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-envoy\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.213201 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.213192 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-token\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.213530 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.213232 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-credential-socket\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.213530 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.213420 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sslw\" (UniqueName: \"kubernetes.io/projected/a5f9b52d-c597-442c-bdc4-28566eed7ba6-kube-api-access-6sslw\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.213633 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.213539 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-workload-socket\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.213766 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.213691 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-credential-socket\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.213840 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.213808 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-workload-certs\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.213840 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.213824 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istiod-ca-cert\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.213952 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.213905 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-data\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.215549 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.215532 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-podinfo\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.215773 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.215740 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-envoy\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.224026 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.223992 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sslw\" (UniqueName: \"kubernetes.io/projected/a5f9b52d-c597-442c-bdc4-28566eed7ba6-kube-api-access-6sslw\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.224444 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.224426 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-token\") pod \"e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.349869 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.349743 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:57.487786 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.487740 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97"] Jul 15 20:13:57.488694 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:13:57.488671 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f9b52d_c597_442c_bdc4_28566eed7ba6.slice/crio-f7e4a734fb2575a1cf7342d0028a6220e44908ac7c1b4a7976204bc03a0f5fd0 WatchSource:0}: Error finding container f7e4a734fb2575a1cf7342d0028a6220e44908ac7c1b4a7976204bc03a0f5fd0: Status 404 returned error can't find the container with id f7e4a734fb2575a1cf7342d0028a6220e44908ac7c1b4a7976204bc03a0f5fd0 Jul 15 20:13:57.490780 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.490730 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:13:57.490849 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.490838 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:13:57.490892 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:57.490866 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:13:58.111957 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:58.111912 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" event={"ID":"a5f9b52d-c597-442c-bdc4-28566eed7ba6","Type":"ContainerStarted","Data":"d7fc8226b4872327787bedc4dfb17cd30189a435aabe1f56075ccee4c7f3f19e"} Jul 15 20:13:58.112439 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:58.112021 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" event={"ID":"a5f9b52d-c597-442c-bdc4-28566eed7ba6","Type":"ContainerStarted","Data":"f7e4a734fb2575a1cf7342d0028a6220e44908ac7c1b4a7976204bc03a0f5fd0"} Jul 15 20:13:58.138081 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:58.138027 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" podStartSLOduration=1.138009562 podStartE2EDuration="1.138009562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:13:58.134653143 +0000 UTC m=+2152.379159603" watchObservedRunningTime="2026-07-15 20:13:58.138009562 +0000 UTC m=+2152.382516023" Jul 15 20:13:58.350823 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:58.350776 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:58.355760 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:58.355716 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:59.118379 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:59.118348 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:13:59.119428 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:13:59.119406 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:14:00.168337 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.168292 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:14:00.240184 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.240153 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw"] Jul 15 20:14:00.240554 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.240540 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="00f1964a-ad10-4c9b-ad08-7231629227a7" containerName="istio-proxy" Jul 15 20:14:00.242654 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.242636 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" Jul 15 20:14:00.246382 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.246359 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8950fff5-serving-cert\"" Jul 15 20:14:00.246382 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.246383 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 20:14:00.246585 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.246515 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 20:14:00.248886 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.248871 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c5cgj\"" Jul 15 20:14:00.256898 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.256876 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw"] Jul 15 20:14:00.284530 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.284494 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:14:00.341338 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.341293 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8baab09d-9b19-413c-af60-8bb4ffc1194f-maas-api-tls\") pod \"maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw\" (UID: \"8baab09d-9b19-413c-af60-8bb4ffc1194f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" Jul 15 20:14:00.341526 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.341378 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8baab09d-9b19-413c-af60-8bb4ffc1194f-openshift-service-ca\") pod \"maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw\" (UID: \"8baab09d-9b19-413c-af60-8bb4ffc1194f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" Jul 15 20:14:00.341526 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.341430 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfvw5\" (UniqueName: \"kubernetes.io/projected/8baab09d-9b19-413c-af60-8bb4ffc1194f-kube-api-access-qfvw5\") pod \"maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw\" (UID: \"8baab09d-9b19-413c-af60-8bb4ffc1194f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" Jul 15 20:14:00.390936 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.390899 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:14:00.442271 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.442183 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8baab09d-9b19-413c-af60-8bb4ffc1194f-maas-api-tls\") pod \"maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw\" (UID: \"8baab09d-9b19-413c-af60-8bb4ffc1194f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" Jul 15 20:14:00.442271 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.442221 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8baab09d-9b19-413c-af60-8bb4ffc1194f-openshift-service-ca\") pod \"maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw\" (UID: \"8baab09d-9b19-413c-af60-8bb4ffc1194f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" Jul 15 20:14:00.442479 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:14:00.442383 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-serving-cert: secret "maas-api-e2e-mt-8950fff5-serving-cert" not found Jul 15 20:14:00.442479 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.442438 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfvw5\" (UniqueName: \"kubernetes.io/projected/8baab09d-9b19-413c-af60-8bb4ffc1194f-kube-api-access-qfvw5\") pod \"maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw\" (UID: \"8baab09d-9b19-413c-af60-8bb4ffc1194f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" Jul 15 20:14:00.442479 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:14:00.442446 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8baab09d-9b19-413c-af60-8bb4ffc1194f-maas-api-tls podName:8baab09d-9b19-413c-af60-8bb4ffc1194f nodeName:}" failed. No retries permitted until 2026-07-15 20:14:00.94242948 +0000 UTC m=+2155.186935918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8baab09d-9b19-413c-af60-8bb4ffc1194f-maas-api-tls") pod "maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" (UID: "8baab09d-9b19-413c-af60-8bb4ffc1194f") : secret "maas-api-e2e-mt-8950fff5-serving-cert" not found Jul 15 20:14:00.442918 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.442900 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8baab09d-9b19-413c-af60-8bb4ffc1194f-openshift-service-ca\") pod \"maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw\" (UID: \"8baab09d-9b19-413c-af60-8bb4ffc1194f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" Jul 15 20:14:00.459086 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.459058 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfvw5\" (UniqueName: \"kubernetes.io/projected/8baab09d-9b19-413c-af60-8bb4ffc1194f-kube-api-access-qfvw5\") pod \"maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw\" (UID: \"8baab09d-9b19-413c-af60-8bb4ffc1194f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" Jul 15 20:14:00.931135 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.931101 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:14:00.948427 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.948395 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8baab09d-9b19-413c-af60-8bb4ffc1194f-maas-api-tls\") pod \"maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw\" (UID: \"8baab09d-9b19-413c-af60-8bb4ffc1194f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" Jul 15 20:14:00.951006 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:00.950982 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8baab09d-9b19-413c-af60-8bb4ffc1194f-maas-api-tls\") pod \"maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw\" (UID: \"8baab09d-9b19-413c-af60-8bb4ffc1194f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" Jul 15 20:14:01.035949 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:01.035908 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:14:01.156988 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:01.156957 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" Jul 15 20:14:01.310375 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:01.310348 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw"] Jul 15 20:14:01.312124 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:14:01.312096 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8baab09d_9b19_413c_af60_8bb4ffc1194f.slice/crio-ccadec0da94afb6755a83eba4038a6e8ea05eb5c3fa8f8a7308de34f1bf50db6 WatchSource:0}: Error finding container ccadec0da94afb6755a83eba4038a6e8ea05eb5c3fa8f8a7308de34f1bf50db6: Status 404 returned error can't find the container with id ccadec0da94afb6755a83eba4038a6e8ea05eb5c3fa8f8a7308de34f1bf50db6 Jul 15 20:14:02.136928 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:02.136893 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" event={"ID":"8baab09d-9b19-413c-af60-8bb4ffc1194f","Type":"ContainerStarted","Data":"ccadec0da94afb6755a83eba4038a6e8ea05eb5c3fa8f8a7308de34f1bf50db6"} Jul 15 20:14:03.144372 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:03.144322 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" event={"ID":"8baab09d-9b19-413c-af60-8bb4ffc1194f","Type":"ContainerStarted","Data":"166fadce6244725383a959c330b2699e4f8af891fecb505f5d983c98c48bd41e"} Jul 15 20:14:03.144859 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:03.144397 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" Jul 15 20:14:03.164534 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:03.164487 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" podStartSLOduration=2.242481804 podStartE2EDuration="3.164471732s" totalImagesPullingTime="921.989928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:14:03.163174154 +0000 UTC m=+2157.407680613" watchObservedRunningTime="2026-07-15 20:14:03.164471732 +0000 UTC m=+2157.408978193" Jul 15 20:14:04.589932 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:04.589900 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:14:05.537600 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:05.537568 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:14:06.479794 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:06.479738 2554 scope.go:117] "RemoveContainer" containerID="70554b713492c0c52c6aa39cabcbe4f987814376fab661787b9f4f7af25d233c" Jul 15 20:14:09.161655 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:09.161626 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" Jul 15 20:14:14.119699 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:14.119660 2554 patch_prober.go:28] interesting pod/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 15 20:14:14.120134 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:14.119732 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" podUID="a5f9b52d-c597-442c-bdc4-28566eed7ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 15 20:14:14.214292 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:14.214253 2554 generic.go:356] "Generic (PLEG): container finished" podID="a5f9b52d-c597-442c-bdc4-28566eed7ba6" containerID="d7fc8226b4872327787bedc4dfb17cd30189a435aabe1f56075ccee4c7f3f19e" exitCode=0 Jul 15 20:14:14.214462 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:14.214304 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" event={"ID":"a5f9b52d-c597-442c-bdc4-28566eed7ba6","Type":"ContainerDied","Data":"d7fc8226b4872327787bedc4dfb17cd30189a435aabe1f56075ccee4c7f3f19e"} Jul 15 20:14:14.214714 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:14.214698 2554 scope.go:117] "RemoveContainer" containerID="d7fc8226b4872327787bedc4dfb17cd30189a435aabe1f56075ccee4c7f3f19e" Jul 15 20:14:14.215829 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:14.215794 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:14:14.215918 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:14.215899 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:14:14.215964 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:14.215943 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:14:15.223856 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:15.223800 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" event={"ID":"a5f9b52d-c597-442c-bdc4-28566eed7ba6","Type":"ContainerStarted","Data":"2ba6371928efef2b16b7e60eee3caf55e90aa0621ca5b89ec76e2f396172620b"} Jul 15 20:14:15.350277 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:15.350235 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:14:15.350580 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:15.350554 2554 patch_prober.go:28] interesting pod/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 15 20:14:15.350635 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:15.350606 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" podUID="a5f9b52d-c597-442c-bdc4-28566eed7ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 15 20:14:16.350350 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:16.350309 2554 patch_prober.go:28] interesting pod/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 15 20:14:16.350745 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:16.350372 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" podUID="a5f9b52d-c597-442c-bdc4-28566eed7ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 15 20:14:17.350453 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:17.350412 2554 patch_prober.go:28] interesting pod/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 15 20:14:17.350887 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:17.350471 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" podUID="a5f9b52d-c597-442c-bdc4-28566eed7ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 15 20:14:18.350891 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:18.350853 2554 patch_prober.go:28] interesting pod/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 15 20:14:18.351386 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:18.350918 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" podUID="a5f9b52d-c597-442c-bdc4-28566eed7ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 15 20:14:19.354352 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:19.354319 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:14:19.354805 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:19.354741 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:14:19.355422 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:19.355402 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:14:21.162211 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:21.162169 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:14:24.732091 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:24.732042 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:14:25.394488 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:25.394458 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:14:29.886895 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:29.886854 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:14:33.042842 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:14:33.042807 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:15:00.141538 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:00.141498 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735775-qp6w4"] Jul 15 20:15:00.144264 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:00.144235 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735775-qp6w4" Jul 15 20:15:00.144875 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:00.144854 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8950fff5-29735775-b8sff"] Jul 15 20:15:00.146935 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:00.146914 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8950fff5-29735775-b8sff" Jul 15 20:15:00.147122 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:00.147105 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tkj8f\"" Jul 15 20:15:00.154296 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:00.154260 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735775-qp6w4"] Jul 15 20:15:00.157703 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:00.157679 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8950fff5-29735775-b8sff"] Jul 15 20:15:00.157855 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:00.157838 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735775-qp6w4" Jul 15 20:15:00.165922 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:00.165896 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8950fff5-29735775-b8sff" Jul 15 20:15:00.311917 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:00.311880 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735775-qp6w4"] Jul 15 20:15:00.315051 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:15:00.315024 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14970f6_54c0_4be5_bc2a_3a6f253214e3.slice/crio-cfda466b41f3621bf61ebc373569cec0a50867add27cdc16a9a4f5c6bd183eb4 WatchSource:0}: Error finding container cfda466b41f3621bf61ebc373569cec0a50867add27cdc16a9a4f5c6bd183eb4: Status 404 returned error can't find the container with id cfda466b41f3621bf61ebc373569cec0a50867add27cdc16a9a4f5c6bd183eb4 Jul 15 20:15:00.346628 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:00.346598 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8950fff5-29735775-b8sff"] Jul 15 20:15:00.363073 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:15:00.363042 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc516fd3a_5a48_4821_8905_9f2b1ac9509e.slice/crio-34eaa36dab180e049be3b45917b31295859fe23d9a6bb729d49d690ba362b66a WatchSource:0}: Error finding container 34eaa36dab180e049be3b45917b31295859fe23d9a6bb729d49d690ba362b66a: Status 404 returned error can't find the container with id 34eaa36dab180e049be3b45917b31295859fe23d9a6bb729d49d690ba362b66a Jul 15 20:15:00.484562 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:00.484525 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8950fff5-29735775-b8sff" event={"ID":"c516fd3a-5a48-4821-8905-9f2b1ac9509e","Type":"ContainerStarted","Data":"34eaa36dab180e049be3b45917b31295859fe23d9a6bb729d49d690ba362b66a"} Jul 15 20:15:00.485767 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:00.485729 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735775-qp6w4" event={"ID":"e14970f6-54c0-4be5-bc2a-3a6f253214e3","Type":"ContainerStarted","Data":"cfda466b41f3621bf61ebc373569cec0a50867add27cdc16a9a4f5c6bd183eb4"} Jul 15 20:15:01.493097 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:01.493002 2554 generic.go:356] "Generic (PLEG): container finished" podID="c516fd3a-5a48-4821-8905-9f2b1ac9509e" containerID="effc5318df75df1a1fe8a9d3fe5ec4a9f94f2f75de1b41a6e16a23bb597ff2bf" exitCode=0 Jul 15 20:15:01.493097 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:01.493084 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8950fff5-29735775-b8sff" event={"ID":"c516fd3a-5a48-4821-8905-9f2b1ac9509e","Type":"ContainerDied","Data":"effc5318df75df1a1fe8a9d3fe5ec4a9f94f2f75de1b41a6e16a23bb597ff2bf"} Jul 15 20:15:01.494667 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:01.494645 2554 generic.go:356] "Generic (PLEG): container finished" podID="e14970f6-54c0-4be5-bc2a-3a6f253214e3" containerID="e8642042bb5324c41b14d849e99798949a66a29f0becaae8151ae28e678ad7fe" exitCode=0 Jul 15 20:15:01.494798 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:01.494671 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735775-qp6w4" event={"ID":"e14970f6-54c0-4be5-bc2a-3a6f253214e3","Type":"ContainerDied","Data":"e8642042bb5324c41b14d849e99798949a66a29f0becaae8151ae28e678ad7fe"} Jul 15 20:15:02.640781 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:02.640737 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8950fff5-29735775-b8sff" Jul 15 20:15:02.670905 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:02.670880 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735775-qp6w4" Jul 15 20:15:03.506606 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:03.506573 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8950fff5-29735775-b8sff" Jul 15 20:15:03.506816 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:03.506574 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8950fff5-29735775-b8sff" event={"ID":"c516fd3a-5a48-4821-8905-9f2b1ac9509e","Type":"ContainerDied","Data":"34eaa36dab180e049be3b45917b31295859fe23d9a6bb729d49d690ba362b66a"} Jul 15 20:15:03.506816 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:03.506701 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34eaa36dab180e049be3b45917b31295859fe23d9a6bb729d49d690ba362b66a" Jul 15 20:15:03.508066 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:03.508036 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735775-qp6w4" Jul 15 20:15:03.508178 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:03.508089 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735775-qp6w4" event={"ID":"e14970f6-54c0-4be5-bc2a-3a6f253214e3","Type":"ContainerDied","Data":"cfda466b41f3621bf61ebc373569cec0a50867add27cdc16a9a4f5c6bd183eb4"} Jul 15 20:15:03.508178 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:03.508128 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfda466b41f3621bf61ebc373569cec0a50867add27cdc16a9a4f5c6bd183eb4" Jul 15 20:15:06.514022 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:06.513994 2554 scope.go:117] "RemoveContainer" containerID="1dbbff0c81e1f41933e0e6d214f1487d4ed1332b909c6a186cc5a469febaff8f" Jul 15 20:15:10.573933 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:10.573896 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:15:13.712813 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:13.712778 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8950fff5-6j4dw"] Jul 15 20:15:13.713204 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:13.713145 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c516fd3a-5a48-4821-8905-9f2b1ac9509e" containerName="cleanup" Jul 15 20:15:13.713204 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:13.713156 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c516fd3a-5a48-4821-8905-9f2b1ac9509e" containerName="cleanup" Jul 15 20:15:13.713204 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:13.713163 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e14970f6-54c0-4be5-bc2a-3a6f253214e3" containerName="cleanup" Jul 15 20:15:13.713204 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:13.713169 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e14970f6-54c0-4be5-bc2a-3a6f253214e3" containerName="cleanup" Jul 15 20:15:13.713343 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:13.713222 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c516fd3a-5a48-4821-8905-9f2b1ac9509e" containerName="cleanup" Jul 15 20:15:13.713343 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:13.713231 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e14970f6-54c0-4be5-bc2a-3a6f253214e3" containerName="cleanup" Jul 15 20:15:13.715251 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:13.715233 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8950fff5-6j4dw" Jul 15 20:15:13.718299 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:13.718275 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tkj8f\"" Jul 15 20:15:13.734276 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:13.734248 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8950fff5-6j4dw"] Jul 15 20:15:13.797594 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:13.797562 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/331d99af-d49b-41a8-bf24-3ec0b87b3145-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8950fff5-6j4dw\" (UID: \"331d99af-d49b-41a8-bf24-3ec0b87b3145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8950fff5-6j4dw" Jul 15 20:15:13.898423 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:13.898384 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/331d99af-d49b-41a8-bf24-3ec0b87b3145-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8950fff5-6j4dw\" (UID: \"331d99af-d49b-41a8-bf24-3ec0b87b3145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8950fff5-6j4dw" Jul 15 20:15:13.899031 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:13.899013 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/331d99af-d49b-41a8-bf24-3ec0b87b3145-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8950fff5-6j4dw\" (UID: \"331d99af-d49b-41a8-bf24-3ec0b87b3145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8950fff5-6j4dw" Jul 15 20:15:14.026232 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:14.026140 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8950fff5-6j4dw" Jul 15 20:15:14.173104 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:14.173076 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8950fff5-6j4dw"] Jul 15 20:15:14.176141 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:15:14.176113 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod331d99af_d49b_41a8_bf24_3ec0b87b3145.slice/crio-8bbd0fd2c228903d2b8ce938448b49f96f45f0c0ab29d22c120cfc7cf61e4114 WatchSource:0}: Error finding container 8bbd0fd2c228903d2b8ce938448b49f96f45f0c0ab29d22c120cfc7cf61e4114: Status 404 returned error can't find the container with id 8bbd0fd2c228903d2b8ce938448b49f96f45f0c0ab29d22c120cfc7cf61e4114 Jul 15 20:15:14.571337 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:14.571293 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8950fff5-6j4dw" event={"ID":"331d99af-d49b-41a8-bf24-3ec0b87b3145","Type":"ContainerStarted","Data":"c0a0da53e999fec915e2a66de257a51ebb880c1e395b64f7e57509c511f0dbec"} Jul 15 20:15:14.571509 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:14.571355 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8950fff5-6j4dw" event={"ID":"331d99af-d49b-41a8-bf24-3ec0b87b3145","Type":"ContainerStarted","Data":"8bbd0fd2c228903d2b8ce938448b49f96f45f0c0ab29d22c120cfc7cf61e4114"} Jul 15 20:15:14.594233 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:14.594130 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8950fff5-6j4dw" podStartSLOduration=1.594114405 podStartE2EDuration="1.594114405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:15:14.592338256 +0000 UTC m=+2228.836844717" watchObservedRunningTime="2026-07-15 20:15:14.594114405 +0000 UTC m=+2228.838620894" Jul 15 20:15:15.578061 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:15.578028 2554 generic.go:356] "Generic (PLEG): container finished" podID="331d99af-d49b-41a8-bf24-3ec0b87b3145" containerID="c0a0da53e999fec915e2a66de257a51ebb880c1e395b64f7e57509c511f0dbec" exitCode=0 Jul 15 20:15:15.578453 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:15.578095 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8950fff5-6j4dw" event={"ID":"331d99af-d49b-41a8-bf24-3ec0b87b3145","Type":"ContainerDied","Data":"c0a0da53e999fec915e2a66de257a51ebb880c1e395b64f7e57509c511f0dbec"} Jul 15 20:15:16.724920 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:16.724890 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8950fff5-6j4dw" Jul 15 20:15:16.826345 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:16.826304 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/331d99af-d49b-41a8-bf24-3ec0b87b3145-maas-api-service-ca\") pod \"331d99af-d49b-41a8-bf24-3ec0b87b3145\" (UID: \"331d99af-d49b-41a8-bf24-3ec0b87b3145\") " Jul 15 20:15:16.826667 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:16.826644 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/331d99af-d49b-41a8-bf24-3ec0b87b3145-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "331d99af-d49b-41a8-bf24-3ec0b87b3145" (UID: "331d99af-d49b-41a8-bf24-3ec0b87b3145"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:15:16.927522 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:16.927491 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/331d99af-d49b-41a8-bf24-3ec0b87b3145-maas-api-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:15:17.590076 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:17.590050 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8950fff5-6j4dw" Jul 15 20:15:17.590255 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:17.590096 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8950fff5-6j4dw" event={"ID":"331d99af-d49b-41a8-bf24-3ec0b87b3145","Type":"ContainerDied","Data":"8bbd0fd2c228903d2b8ce938448b49f96f45f0c0ab29d22c120cfc7cf61e4114"} Jul 15 20:15:17.590255 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:17.590138 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bbd0fd2c228903d2b8ce938448b49f96f45f0c0ab29d22c120cfc7cf61e4114" Jul 15 20:15:24.268218 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.268182 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw"] Jul 15 20:15:24.268702 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.268423 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" podUID="8baab09d-9b19-413c-af60-8bb4ffc1194f" containerName="maas-api" containerID="cri-o://166fadce6244725383a959c330b2699e4f8af891fecb505f5d983c98c48bd41e" gracePeriod=30 Jul 15 20:15:24.303551 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.303508 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8950fff5-29735775-b8sff"] Jul 15 20:15:24.306395 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.306367 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:15:24.307111 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.307091 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8950fff5-29735775-b8sff"] Jul 15 20:15:24.525017 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.524955 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" Jul 15 20:15:24.628309 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.628271 2554 generic.go:356] "Generic (PLEG): container finished" podID="8baab09d-9b19-413c-af60-8bb4ffc1194f" containerID="166fadce6244725383a959c330b2699e4f8af891fecb505f5d983c98c48bd41e" exitCode=0 Jul 15 20:15:24.628480 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.628323 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" Jul 15 20:15:24.628480 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.628361 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" event={"ID":"8baab09d-9b19-413c-af60-8bb4ffc1194f","Type":"ContainerDied","Data":"166fadce6244725383a959c330b2699e4f8af891fecb505f5d983c98c48bd41e"} Jul 15 20:15:24.628480 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.628420 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw" event={"ID":"8baab09d-9b19-413c-af60-8bb4ffc1194f","Type":"ContainerDied","Data":"ccadec0da94afb6755a83eba4038a6e8ea05eb5c3fa8f8a7308de34f1bf50db6"} Jul 15 20:15:24.628480 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.628446 2554 scope.go:117] "RemoveContainer" containerID="166fadce6244725383a959c330b2699e4f8af891fecb505f5d983c98c48bd41e" Jul 15 20:15:24.638037 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.638016 2554 scope.go:117] "RemoveContainer" containerID="166fadce6244725383a959c330b2699e4f8af891fecb505f5d983c98c48bd41e" Jul 15 20:15:24.638348 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:15:24.638328 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"166fadce6244725383a959c330b2699e4f8af891fecb505f5d983c98c48bd41e\": container with ID starting with 166fadce6244725383a959c330b2699e4f8af891fecb505f5d983c98c48bd41e not found: ID does not exist" containerID="166fadce6244725383a959c330b2699e4f8af891fecb505f5d983c98c48bd41e" Jul 15 20:15:24.638401 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.638359 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"166fadce6244725383a959c330b2699e4f8af891fecb505f5d983c98c48bd41e"} err="failed to get container status \"166fadce6244725383a959c330b2699e4f8af891fecb505f5d983c98c48bd41e\": rpc error: code = NotFound desc = could not find container \"166fadce6244725383a959c330b2699e4f8af891fecb505f5d983c98c48bd41e\": container with ID starting with 166fadce6244725383a959c330b2699e4f8af891fecb505f5d983c98c48bd41e not found: ID does not exist" Jul 15 20:15:24.698563 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.698534 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfvw5\" (UniqueName: \"kubernetes.io/projected/8baab09d-9b19-413c-af60-8bb4ffc1194f-kube-api-access-qfvw5\") pod \"8baab09d-9b19-413c-af60-8bb4ffc1194f\" (UID: \"8baab09d-9b19-413c-af60-8bb4ffc1194f\") " Jul 15 20:15:24.698733 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.698627 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8baab09d-9b19-413c-af60-8bb4ffc1194f-maas-api-tls\") pod \"8baab09d-9b19-413c-af60-8bb4ffc1194f\" (UID: \"8baab09d-9b19-413c-af60-8bb4ffc1194f\") " Jul 15 20:15:24.698820 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.698727 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8baab09d-9b19-413c-af60-8bb4ffc1194f-openshift-service-ca\") pod \"8baab09d-9b19-413c-af60-8bb4ffc1194f\" (UID: \"8baab09d-9b19-413c-af60-8bb4ffc1194f\") " Jul 15 20:15:24.699106 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.699084 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8baab09d-9b19-413c-af60-8bb4ffc1194f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8baab09d-9b19-413c-af60-8bb4ffc1194f" (UID: "8baab09d-9b19-413c-af60-8bb4ffc1194f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:15:24.700853 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.700811 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8baab09d-9b19-413c-af60-8bb4ffc1194f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8baab09d-9b19-413c-af60-8bb4ffc1194f" (UID: "8baab09d-9b19-413c-af60-8bb4ffc1194f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:15:24.700948 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.700859 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8baab09d-9b19-413c-af60-8bb4ffc1194f-kube-api-access-qfvw5" (OuterVolumeSpecName: "kube-api-access-qfvw5") pod "8baab09d-9b19-413c-af60-8bb4ffc1194f" (UID: "8baab09d-9b19-413c-af60-8bb4ffc1194f"). InnerVolumeSpecName "kube-api-access-qfvw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:15:24.800422 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.800331 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8baab09d-9b19-413c-af60-8bb4ffc1194f-maas-api-tls\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:15:24.800422 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.800362 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8baab09d-9b19-413c-af60-8bb4ffc1194f-openshift-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:15:24.800422 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.800374 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfvw5\" (UniqueName: \"kubernetes.io/projected/8baab09d-9b19-413c-af60-8bb4ffc1194f-kube-api-access-qfvw5\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:15:24.955464 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.955429 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw"] Jul 15 20:15:24.962537 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:24.962506 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8950fff5-64dcdbc95f-ldfqw"] Jul 15 20:15:26.239290 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:26.239252 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8baab09d-9b19-413c-af60-8bb4ffc1194f" path="/var/lib/kubelet/pods/8baab09d-9b19-413c-af60-8bb4ffc1194f/volumes" Jul 15 20:15:26.239675 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:26.239553 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c516fd3a-5a48-4821-8905-9f2b1ac9509e" path="/var/lib/kubelet/pods/c516fd3a-5a48-4821-8905-9f2b1ac9509e/volumes" Jul 15 20:15:27.841085 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:27.841003 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:15:28.822923 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:28.822892 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:15:33.883473 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:33.883434 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:15:51.858486 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:15:51.858452 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:04.292252 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:04.292211 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:06.535459 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:06.535427 2554 scope.go:117] "RemoveContainer" containerID="97958d8a76d04d7ffdc28bdeae751a6e1a4210cd6c4413b1c8d98ca25390cb05" Jul 15 20:16:10.197229 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:10.197195 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:11.062198 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:11.062156 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:12.045399 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:12.045362 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:18.466077 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:18.466041 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:18.563403 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:18.563367 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:21.318520 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:21.318484 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97"] Jul 15 20:16:21.318988 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:21.318695 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" podUID="a5f9b52d-c597-442c-bdc4-28566eed7ba6" containerName="istio-proxy" containerID="cri-o://2ba6371928efef2b16b7e60eee3caf55e90aa0621ca5b89ec76e2f396172620b" gracePeriod=30 Jul 15 20:16:21.333897 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:21.333866 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:21.566416 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:21.566380 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:22.780668 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:22.780637 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:22.878370 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:22.878338 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:22.992164 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:22.992134 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:23.768004 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:23.767966 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:23.874206 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:23.874172 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:23.988561 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:23.988523 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:25.544972 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:25.544931 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:26.450412 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.450373 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:16:26.575175 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.575149 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:16:26.710968 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.710937 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-token\") pod \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " Jul 15 20:16:26.711164 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.710996 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-workload-socket\") pod \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " Jul 15 20:16:26.711164 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.711020 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-podinfo\") pod \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " Jul 15 20:16:26.711164 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.711046 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-data\") pod \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " Jul 15 20:16:26.711164 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.711087 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-envoy\") pod \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " Jul 15 20:16:26.711164 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.711101 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-credential-socket\") pod \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " Jul 15 20:16:26.711164 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.711132 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-workload-certs\") pod \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " Jul 15 20:16:26.711497 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.711167 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sslw\" (UniqueName: \"kubernetes.io/projected/a5f9b52d-c597-442c-bdc4-28566eed7ba6-kube-api-access-6sslw\") pod \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " Jul 15 20:16:26.711497 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.711193 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istiod-ca-cert\") pod \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\" (UID: \"a5f9b52d-c597-442c-bdc4-28566eed7ba6\") " Jul 15 20:16:26.711597 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.711488 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a5f9b52d-c597-442c-bdc4-28566eed7ba6" (UID: "a5f9b52d-c597-442c-bdc4-28566eed7ba6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:16:26.711597 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.711535 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a5f9b52d-c597-442c-bdc4-28566eed7ba6" (UID: "a5f9b52d-c597-442c-bdc4-28566eed7ba6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:16:26.711713 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.711655 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a5f9b52d-c597-442c-bdc4-28566eed7ba6" (UID: "a5f9b52d-c597-442c-bdc4-28566eed7ba6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:16:26.711973 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.711830 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a5f9b52d-c597-442c-bdc4-28566eed7ba6" (UID: "a5f9b52d-c597-442c-bdc4-28566eed7ba6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:16:26.712446 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.712415 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-data" (OuterVolumeSpecName: "istio-data") pod "a5f9b52d-c597-442c-bdc4-28566eed7ba6" (UID: "a5f9b52d-c597-442c-bdc4-28566eed7ba6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:16:26.713504 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.713470 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a5f9b52d-c597-442c-bdc4-28566eed7ba6" (UID: "a5f9b52d-c597-442c-bdc4-28566eed7ba6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:16:26.713599 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.713529 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-token" (OuterVolumeSpecName: "istio-token") pod "a5f9b52d-c597-442c-bdc4-28566eed7ba6" (UID: "a5f9b52d-c597-442c-bdc4-28566eed7ba6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:16:26.713599 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.713578 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f9b52d-c597-442c-bdc4-28566eed7ba6-kube-api-access-6sslw" (OuterVolumeSpecName: "kube-api-access-6sslw") pod "a5f9b52d-c597-442c-bdc4-28566eed7ba6" (UID: "a5f9b52d-c597-442c-bdc4-28566eed7ba6"). InnerVolumeSpecName "kube-api-access-6sslw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:16:26.713599 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.713582 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a5f9b52d-c597-442c-bdc4-28566eed7ba6" (UID: "a5f9b52d-c597-442c-bdc4-28566eed7ba6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:16:26.812687 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.812650 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-podinfo\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:16:26.812687 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.812678 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-data\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:16:26.812687 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.812690 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-envoy\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:16:26.812687 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.812697 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-credential-socket\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:16:26.812986 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.812706 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-workload-certs\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:16:26.812986 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.812714 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sslw\" (UniqueName: \"kubernetes.io/projected/a5f9b52d-c597-442c-bdc4-28566eed7ba6-kube-api-access-6sslw\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:16:26.812986 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.812722 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istiod-ca-cert\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:16:26.812986 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.812729 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f9b52d-c597-442c-bdc4-28566eed7ba6-istio-token\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:16:26.812986 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.812737 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b52d-c597-442c-bdc4-28566eed7ba6-workload-socket\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:16:26.972113 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.972029 2554 generic.go:356] "Generic (PLEG): container finished" podID="a5f9b52d-c597-442c-bdc4-28566eed7ba6" containerID="2ba6371928efef2b16b7e60eee3caf55e90aa0621ca5b89ec76e2f396172620b" exitCode=0 Jul 15 20:16:26.972113 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.972075 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" Jul 15 20:16:26.972294 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.972115 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" event={"ID":"a5f9b52d-c597-442c-bdc4-28566eed7ba6","Type":"ContainerDied","Data":"2ba6371928efef2b16b7e60eee3caf55e90aa0621ca5b89ec76e2f396172620b"} Jul 15 20:16:26.972294 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.972172 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97" event={"ID":"a5f9b52d-c597-442c-bdc4-28566eed7ba6","Type":"ContainerDied","Data":"f7e4a734fb2575a1cf7342d0028a6220e44908ac7c1b4a7976204bc03a0f5fd0"} Jul 15 20:16:26.972294 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.972194 2554 scope.go:117] "RemoveContainer" containerID="2ba6371928efef2b16b7e60eee3caf55e90aa0621ca5b89ec76e2f396172620b" Jul 15 20:16:26.982049 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.982028 2554 scope.go:117] "RemoveContainer" containerID="d7fc8226b4872327787bedc4dfb17cd30189a435aabe1f56075ccee4c7f3f19e" Jul 15 20:16:26.990511 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.990489 2554 scope.go:117] "RemoveContainer" containerID="2ba6371928efef2b16b7e60eee3caf55e90aa0621ca5b89ec76e2f396172620b" Jul 15 20:16:26.990806 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:16:26.990786 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba6371928efef2b16b7e60eee3caf55e90aa0621ca5b89ec76e2f396172620b\": container with ID starting with 2ba6371928efef2b16b7e60eee3caf55e90aa0621ca5b89ec76e2f396172620b not found: ID does not exist" containerID="2ba6371928efef2b16b7e60eee3caf55e90aa0621ca5b89ec76e2f396172620b" Jul 15 20:16:26.990882 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.990821 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba6371928efef2b16b7e60eee3caf55e90aa0621ca5b89ec76e2f396172620b"} err="failed to get container status \"2ba6371928efef2b16b7e60eee3caf55e90aa0621ca5b89ec76e2f396172620b\": rpc error: code = NotFound desc = could not find container \"2ba6371928efef2b16b7e60eee3caf55e90aa0621ca5b89ec76e2f396172620b\": container with ID starting with 2ba6371928efef2b16b7e60eee3caf55e90aa0621ca5b89ec76e2f396172620b not found: ID does not exist" Jul 15 20:16:26.990882 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.990848 2554 scope.go:117] "RemoveContainer" containerID="d7fc8226b4872327787bedc4dfb17cd30189a435aabe1f56075ccee4c7f3f19e" Jul 15 20:16:26.991125 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:16:26.991107 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7fc8226b4872327787bedc4dfb17cd30189a435aabe1f56075ccee4c7f3f19e\": container with ID starting with d7fc8226b4872327787bedc4dfb17cd30189a435aabe1f56075ccee4c7f3f19e not found: ID does not exist" containerID="d7fc8226b4872327787bedc4dfb17cd30189a435aabe1f56075ccee4c7f3f19e" Jul 15 20:16:26.991165 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:26.991131 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7fc8226b4872327787bedc4dfb17cd30189a435aabe1f56075ccee4c7f3f19e"} err="failed to get container status \"d7fc8226b4872327787bedc4dfb17cd30189a435aabe1f56075ccee4c7f3f19e\": rpc error: code = NotFound desc = could not find container \"d7fc8226b4872327787bedc4dfb17cd30189a435aabe1f56075ccee4c7f3f19e\": container with ID starting with d7fc8226b4872327787bedc4dfb17cd30189a435aabe1f56075ccee4c7f3f19e not found: ID does not exist" Jul 15 20:16:27.006800 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:27.006770 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97"] Jul 15 20:16:27.021078 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:27.021054 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8950fff5-openshift-default-76c457f76c-lpn97"] Jul 15 20:16:28.238840 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:28.238805 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5f9b52d-c597-442c-bdc4-28566eed7ba6" path="/var/lib/kubelet/pods/a5f9b52d-c597-442c-bdc4-28566eed7ba6/volumes" Jul 15 20:16:32.453136 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:32.453100 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:32.538872 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:32.538825 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:33.146079 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:33.146040 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:38.749054 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:38.749017 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:16:39.383396 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:39.383359 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:39.487250 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:39.487213 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:39.591492 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:39.591449 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:50.141098 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:50.141063 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:50.481892 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:16:50.481856 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:11.449114 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:11.449078 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:17.414714 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:17.414664 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:17.517946 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:17.517909 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:17.749269 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:17.749234 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:18.035376 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.035300 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p"] Jul 15 20:17:18.035656 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.035644 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5f9b52d-c597-442c-bdc4-28566eed7ba6" containerName="istio-proxy" Jul 15 20:17:18.035699 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.035658 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5f9b52d-c597-442c-bdc4-28566eed7ba6" containerName="istio-proxy" Jul 15 20:17:18.035699 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.035671 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8baab09d-9b19-413c-af60-8bb4ffc1194f" containerName="maas-api" Jul 15 20:17:18.035699 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.035677 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8baab09d-9b19-413c-af60-8bb4ffc1194f" containerName="maas-api" Jul 15 20:17:18.035699 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.035693 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="331d99af-d49b-41a8-bf24-3ec0b87b3145" containerName="revoke-keys" Jul 15 20:17:18.035699 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.035699 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="331d99af-d49b-41a8-bf24-3ec0b87b3145" containerName="revoke-keys" Jul 15 20:17:18.035877 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.035709 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5f9b52d-c597-442c-bdc4-28566eed7ba6" containerName="istio-proxy" Jul 15 20:17:18.035877 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.035715 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5f9b52d-c597-442c-bdc4-28566eed7ba6" containerName="istio-proxy" Jul 15 20:17:18.035877 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.035779 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5f9b52d-c597-442c-bdc4-28566eed7ba6" containerName="istio-proxy" Jul 15 20:17:18.035877 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.035788 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="331d99af-d49b-41a8-bf24-3ec0b87b3145" containerName="revoke-keys" Jul 15 20:17:18.035877 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.035795 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5f9b52d-c597-442c-bdc4-28566eed7ba6" containerName="istio-proxy" Jul 15 20:17:18.035877 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.035801 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8baab09d-9b19-413c-af60-8bb4ffc1194f" containerName="maas-api" Jul 15 20:17:18.038702 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.038685 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.041372 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.041354 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c923f1-openshift-default-dockercfg-sxdq6\"" Jul 15 20:17:18.052704 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.052679 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p"] Jul 15 20:17:18.061611 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.061574 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:18.149935 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.149901 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-token\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.150103 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.149950 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-workload-socket\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.150103 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.149984 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-workload-certs\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.150103 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.150001 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5742ff-d4c8-4116-b01f-60883b57d9df-istiod-ca-cert\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.150103 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.150022 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-envoy\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.150103 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.150053 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-podinfo\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.150272 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.150109 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-credential-socket\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.150272 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.150152 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-data\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.150272 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.150175 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-452dc\" (UniqueName: \"kubernetes.io/projected/7e5742ff-d4c8-4116-b01f-60883b57d9df-kube-api-access-452dc\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.250933 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.250896 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-workload-certs\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.250933 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.250933 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5742ff-d4c8-4116-b01f-60883b57d9df-istiod-ca-cert\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.251181 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.250955 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-envoy\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.251181 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.250983 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-podinfo\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.251181 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.251016 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-credential-socket\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.251181 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.251051 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-data\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.251181 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.251076 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-452dc\" (UniqueName: \"kubernetes.io/projected/7e5742ff-d4c8-4116-b01f-60883b57d9df-kube-api-access-452dc\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.251459 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.251194 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-token\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.251459 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.251267 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-workload-socket\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.251569 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.251515 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-credential-socket\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.251569 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.251527 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-data\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.251672 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.251597 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-workload-socket\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.251672 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.251638 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-workload-certs\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.251787 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.251743 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5742ff-d4c8-4116-b01f-60883b57d9df-istiod-ca-cert\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.253927 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.253902 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-podinfo\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.254085 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.254066 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-envoy\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.262869 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.262841 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-token\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.263024 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.262931 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-452dc\" (UniqueName: \"kubernetes.io/projected/7e5742ff-d4c8-4116-b01f-60883b57d9df-kube-api-access-452dc\") pod \"e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.349447 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.349351 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:18.498947 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.498917 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p"] Jul 15 20:17:18.500130 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:17:18.500099 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5742ff_d4c8_4116_b01f_60883b57d9df.slice/crio-7466fc84288ea3c6b429cd4183ca14d9fb10a48cafa5156b77acdb1614e903f7 WatchSource:0}: Error finding container 7466fc84288ea3c6b429cd4183ca14d9fb10a48cafa5156b77acdb1614e903f7: Status 404 returned error can't find the container with id 7466fc84288ea3c6b429cd4183ca14d9fb10a48cafa5156b77acdb1614e903f7 Jul 15 20:17:18.502279 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.502249 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:17:18.502361 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.502314 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:17:18.502361 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.502340 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:17:18.774876 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:18.774836 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:19.272630 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:19.272596 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" event={"ID":"7e5742ff-d4c8-4116-b01f-60883b57d9df","Type":"ContainerStarted","Data":"3bb8068676edc54483a4e253355e97e7b92937c7c85d3c4a000c1235d40f334e"} Jul 15 20:17:19.272815 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:19.272651 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" event={"ID":"7e5742ff-d4c8-4116-b01f-60883b57d9df","Type":"ContainerStarted","Data":"7466fc84288ea3c6b429cd4183ca14d9fb10a48cafa5156b77acdb1614e903f7"} Jul 15 20:17:19.303831 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:19.303784 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" podStartSLOduration=1.303768263 podStartE2EDuration="1.303768263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:17:19.301240114 +0000 UTC m=+2353.545746576" watchObservedRunningTime="2026-07-15 20:17:19.303768263 +0000 UTC m=+2353.548274719" Jul 15 20:17:19.350490 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:19.350453 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:19.355368 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:19.355345 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:20.277888 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:20.277858 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:20.279020 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:20.278995 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:21.290027 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.289984 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:21.360964 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.360931 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts"] Jul 15 20:17:21.365107 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.365076 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" Jul 15 20:17:21.368004 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.367977 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 20:17:21.368133 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.368040 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 20:17:21.368133 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.367992 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c923f1-serving-cert\"" Jul 15 20:17:21.368981 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.368965 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c5cgj\"" Jul 15 20:17:21.386354 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.386325 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts"] Jul 15 20:17:21.397063 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.397029 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:21.471167 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.471128 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:21.484365 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.484324 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73723d2-c98d-4da5-951d-f0b8bdab6a5f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts\" (UID: \"e73723d2-c98d-4da5-951d-f0b8bdab6a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" Jul 15 20:17:21.484574 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.484382 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqcf8\" (UniqueName: \"kubernetes.io/projected/e73723d2-c98d-4da5-951d-f0b8bdab6a5f-kube-api-access-zqcf8\") pod \"maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts\" (UID: \"e73723d2-c98d-4da5-951d-f0b8bdab6a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" Jul 15 20:17:21.484574 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.484558 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e73723d2-c98d-4da5-951d-f0b8bdab6a5f-maas-api-tls\") pod \"maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts\" (UID: \"e73723d2-c98d-4da5-951d-f0b8bdab6a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" Jul 15 20:17:21.572213 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.572119 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:21.585454 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.585427 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e73723d2-c98d-4da5-951d-f0b8bdab6a5f-maas-api-tls\") pod \"maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts\" (UID: \"e73723d2-c98d-4da5-951d-f0b8bdab6a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" Jul 15 20:17:21.585566 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.585463 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73723d2-c98d-4da5-951d-f0b8bdab6a5f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts\" (UID: \"e73723d2-c98d-4da5-951d-f0b8bdab6a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" Jul 15 20:17:21.585566 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.585493 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqcf8\" (UniqueName: \"kubernetes.io/projected/e73723d2-c98d-4da5-951d-f0b8bdab6a5f-kube-api-access-zqcf8\") pod \"maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts\" (UID: \"e73723d2-c98d-4da5-951d-f0b8bdab6a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" Jul 15 20:17:21.586252 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.586230 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73723d2-c98d-4da5-951d-f0b8bdab6a5f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts\" (UID: \"e73723d2-c98d-4da5-951d-f0b8bdab6a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" Jul 15 20:17:21.588019 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.587998 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e73723d2-c98d-4da5-951d-f0b8bdab6a5f-maas-api-tls\") pod \"maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts\" (UID: \"e73723d2-c98d-4da5-951d-f0b8bdab6a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" Jul 15 20:17:21.594465 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.594431 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqcf8\" (UniqueName: \"kubernetes.io/projected/e73723d2-c98d-4da5-951d-f0b8bdab6a5f-kube-api-access-zqcf8\") pod \"maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts\" (UID: \"e73723d2-c98d-4da5-951d-f0b8bdab6a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" Jul 15 20:17:21.676814 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.676775 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" Jul 15 20:17:21.852875 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:21.852675 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts"] Jul 15 20:17:22.080579 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:22.080538 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:22.183251 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:22.183221 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:22.290519 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:22.290480 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" event={"ID":"e73723d2-c98d-4da5-951d-f0b8bdab6a5f","Type":"ContainerStarted","Data":"cafdf4f380d8440b96ffd844b1dc3a51e7e4bdfa78f47f9e87a0aa5b32279962"} Jul 15 20:17:23.298166 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:23.298124 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" event={"ID":"e73723d2-c98d-4da5-951d-f0b8bdab6a5f","Type":"ContainerStarted","Data":"73d94967a608b7da1d9d62690bb049cc236d22737658c8d138fbe96bb63a60cf"} Jul 15 20:17:23.298558 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:23.298200 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" Jul 15 20:17:25.864353 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:25.864310 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:26.185125 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:26.185096 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:26.527693 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:26.527601 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:26.751666 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:26.751636 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:29.035514 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:29.035457 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" podStartSLOduration=7.408035404 podStartE2EDuration="8.035438775s" totalImagesPullingTime="627.403371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:17:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:17:23.320648944 +0000 UTC m=+2357.565155406" watchObservedRunningTime="2026-07-15 20:17:29.035438775 +0000 UTC m=+2363.279945235" Jul 15 20:17:29.036805 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:29.036785 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:17:29.309430 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:29.309335 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" Jul 15 20:17:29.893892 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:29.893860 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:29.992768 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:29.992718 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:30.102060 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:30.102028 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf"] Jul 15 20:17:30.106499 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:30.106478 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" Jul 15 20:17:30.109078 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:30.109055 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a27bd5-serving-cert\"" Jul 15 20:17:30.137492 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:30.137439 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf"] Jul 15 20:17:30.262218 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:30.262186 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a6549a-28c8-4670-ab2e-ff04e5a746cc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf\" (UID: \"a6a6549a-28c8-4670-ab2e-ff04e5a746cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" Jul 15 20:17:30.262218 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:30.262221 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmn57\" (UniqueName: \"kubernetes.io/projected/a6a6549a-28c8-4670-ab2e-ff04e5a746cc-kube-api-access-mmn57\") pod \"maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf\" (UID: \"a6a6549a-28c8-4670-ab2e-ff04e5a746cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" Jul 15 20:17:30.262447 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:30.262249 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a6549a-28c8-4670-ab2e-ff04e5a746cc-maas-api-tls\") pod \"maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf\" (UID: \"a6a6549a-28c8-4670-ab2e-ff04e5a746cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" Jul 15 20:17:30.363217 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:30.363179 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a6549a-28c8-4670-ab2e-ff04e5a746cc-maas-api-tls\") pod \"maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf\" (UID: \"a6a6549a-28c8-4670-ab2e-ff04e5a746cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" Jul 15 20:17:30.363394 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:30.363366 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a6549a-28c8-4670-ab2e-ff04e5a746cc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf\" (UID: \"a6a6549a-28c8-4670-ab2e-ff04e5a746cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" Jul 15 20:17:30.363436 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:30.363396 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmn57\" (UniqueName: \"kubernetes.io/projected/a6a6549a-28c8-4670-ab2e-ff04e5a746cc-kube-api-access-mmn57\") pod \"maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf\" (UID: \"a6a6549a-28c8-4670-ab2e-ff04e5a746cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" Jul 15 20:17:30.364052 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:30.364024 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a6549a-28c8-4670-ab2e-ff04e5a746cc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf\" (UID: \"a6a6549a-28c8-4670-ab2e-ff04e5a746cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" Jul 15 20:17:30.365796 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:30.365774 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a6549a-28c8-4670-ab2e-ff04e5a746cc-maas-api-tls\") pod \"maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf\" (UID: \"a6a6549a-28c8-4670-ab2e-ff04e5a746cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" Jul 15 20:17:30.378794 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:30.378742 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmn57\" (UniqueName: \"kubernetes.io/projected/a6a6549a-28c8-4670-ab2e-ff04e5a746cc-kube-api-access-mmn57\") pod \"maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf\" (UID: \"a6a6549a-28c8-4670-ab2e-ff04e5a746cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" Jul 15 20:17:30.419410 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:30.419353 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" Jul 15 20:17:30.550776 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:30.550311 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:30.645812 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:30.645775 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf"] Jul 15 20:17:30.647090 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:17:30.647064 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a6549a_28c8_4670_ab2e_ff04e5a746cc.slice/crio-b985ca4b5dbf91ab9d1d686795046636f86cfa7c93bd55073af140e9c102029c WatchSource:0}: Error finding container b985ca4b5dbf91ab9d1d686795046636f86cfa7c93bd55073af140e9c102029c: Status 404 returned error can't find the container with id b985ca4b5dbf91ab9d1d686795046636f86cfa7c93bd55073af140e9c102029c Jul 15 20:17:30.677242 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:30.677199 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:31.133544 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:31.133479 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:31.236411 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:31.236309 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:31.349183 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:31.349149 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" event={"ID":"a6a6549a-28c8-4670-ab2e-ff04e5a746cc","Type":"ContainerStarted","Data":"19dc775d7d749352a812609c26f1f5689cbb3a961dad051e3b4fa9fea8fbb3bd"} Jul 15 20:17:31.349365 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:31.349200 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" event={"ID":"a6a6549a-28c8-4670-ab2e-ff04e5a746cc","Type":"ContainerStarted","Data":"b985ca4b5dbf91ab9d1d686795046636f86cfa7c93bd55073af140e9c102029c"} Jul 15 20:17:31.349365 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:31.349288 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" Jul 15 20:17:31.394442 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:31.394384 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" podStartSLOduration=1.117399015 podStartE2EDuration="1.394367267s" totalImagesPullingTime="276.968252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:17:31.39201606 +0000 UTC m=+2365.636522523" watchObservedRunningTime="2026-07-15 20:17:31.394367267 +0000 UTC m=+2365.638873752" Jul 15 20:17:34.427345 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:34.427283 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:36.319954 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:36.319910 2554 patch_prober.go:28] interesting pod/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 20:17:36.320343 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:36.319985 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" podUID="7e5742ff-d4c8-4116-b01f-60883b57d9df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 20:17:37.362298 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:37.362266 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" Jul 15 20:17:37.387957 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:37.387931 2554 generic.go:356] "Generic (PLEG): container finished" podID="7e5742ff-d4c8-4116-b01f-60883b57d9df" containerID="3bb8068676edc54483a4e253355e97e7b92937c7c85d3c4a000c1235d40f334e" exitCode=0 Jul 15 20:17:37.388115 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:37.388012 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" event={"ID":"7e5742ff-d4c8-4116-b01f-60883b57d9df","Type":"ContainerDied","Data":"3bb8068676edc54483a4e253355e97e7b92937c7c85d3c4a000c1235d40f334e"} Jul 15 20:17:37.388407 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:37.388392 2554 scope.go:117] "RemoveContainer" containerID="3bb8068676edc54483a4e253355e97e7b92937c7c85d3c4a000c1235d40f334e" Jul 15 20:17:37.389539 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:37.389513 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:17:37.389612 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:37.389595 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:17:37.389661 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:37.389647 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:17:38.394824 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:38.394780 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" event={"ID":"7e5742ff-d4c8-4116-b01f-60883b57d9df","Type":"ContainerStarted","Data":"1d94b6eb62db21f2ba8473a7011cfcfd806576abedb8a86d51fe884a84fd6dbf"} Jul 15 20:17:39.350481 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:39.350438 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:39.350700 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:39.350678 2554 patch_prober.go:28] interesting pod/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 15 20:17:39.350779 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:39.350726 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" podUID="7e5742ff-d4c8-4116-b01f-60883b57d9df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 15 20:17:40.035490 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:40.035449 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:17:40.350563 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:40.350468 2554 patch_prober.go:28] interesting pod/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 15 20:17:40.350563 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:40.350550 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" podUID="7e5742ff-d4c8-4116-b01f-60883b57d9df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 15 20:17:41.350138 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:41.350096 2554 patch_prober.go:28] interesting pod/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 15 20:17:41.350626 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:41.350169 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" podUID="7e5742ff-d4c8-4116-b01f-60883b57d9df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 15 20:17:42.353807 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:42.353776 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:42.354265 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:42.354068 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:42.354842 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:42.354826 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:17:43.976200 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:43.976152 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:44.078861 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:44.078832 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:51.613893 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:51.613848 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:51.786133 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:51.786098 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:51.886936 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:51.886853 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:17:54.830035 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:17:54.830000 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:18:06.426933 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:18:06.426903 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 20:18:06.431487 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:18:06.431460 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 20:18:06.441094 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:18:06.441073 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 20:18:06.445303 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:18:06.445282 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 20:18:10.135065 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:18:10.135034 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:18:37.071051 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:18:37.070975 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:18:39.393870 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:18:39.393827 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:18:39.495465 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:18:39.495427 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:18:41.633795 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:18:41.633742 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:19:00.335046 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:00.334995 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:19:24.257395 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:24.257365 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:19:25.145844 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:25.145805 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:19:25.243785 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:25.243734 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c923f1-4sv95"] Jul 15 20:19:25.247233 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:25.247211 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c923f1-4sv95" Jul 15 20:19:25.250091 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:25.250070 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tkj8f\"" Jul 15 20:19:25.258648 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:25.258618 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c923f1-4sv95"] Jul 15 20:19:25.275316 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:25.275282 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25d107e-24f5-4953-aad4-1027fe5d1f95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c923f1-4sv95\" (UID: \"d25d107e-24f5-4953-aad4-1027fe5d1f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c923f1-4sv95" Jul 15 20:19:25.376473 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:25.376434 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25d107e-24f5-4953-aad4-1027fe5d1f95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c923f1-4sv95\" (UID: \"d25d107e-24f5-4953-aad4-1027fe5d1f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c923f1-4sv95" Jul 15 20:19:25.377118 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:25.377099 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25d107e-24f5-4953-aad4-1027fe5d1f95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c923f1-4sv95\" (UID: \"d25d107e-24f5-4953-aad4-1027fe5d1f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c923f1-4sv95" Jul 15 20:19:25.559599 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:25.559569 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c923f1-4sv95" Jul 15 20:19:25.699730 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:25.699704 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c923f1-4sv95"] Jul 15 20:19:25.702437 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:19:25.702408 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25d107e_24f5_4953_aad4_1027fe5d1f95.slice/crio-3825a6c44e056ec7d0f012f3e5698e0da5b4f2411252fd35675cf14b70577c93 WatchSource:0}: Error finding container 3825a6c44e056ec7d0f012f3e5698e0da5b4f2411252fd35675cf14b70577c93: Status 404 returned error can't find the container with id 3825a6c44e056ec7d0f012f3e5698e0da5b4f2411252fd35675cf14b70577c93 Jul 15 20:19:25.880147 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:25.880056 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:19:25.979297 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:25.979265 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:19:26.003441 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:26.003409 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c923f1-4sv95" event={"ID":"d25d107e-24f5-4953-aad4-1027fe5d1f95","Type":"ContainerStarted","Data":"3825a6c44e056ec7d0f012f3e5698e0da5b4f2411252fd35675cf14b70577c93"} Jul 15 20:19:27.010803 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:27.010740 2554 generic.go:356] "Generic (PLEG): container finished" podID="d25d107e-24f5-4953-aad4-1027fe5d1f95" containerID="0b6e1880d2a2becba74f73cce50adf3665cde79b448c3a9b8097a65a7af085de" exitCode=0 Jul 15 20:19:27.011190 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:27.010821 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c923f1-4sv95" event={"ID":"d25d107e-24f5-4953-aad4-1027fe5d1f95","Type":"ContainerDied","Data":"0b6e1880d2a2becba74f73cce50adf3665cde79b448c3a9b8097a65a7af085de"} Jul 15 20:19:28.152659 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:28.152635 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c923f1-4sv95" Jul 15 20:19:28.199284 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:28.199251 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25d107e-24f5-4953-aad4-1027fe5d1f95-maas-api-service-ca\") pod \"d25d107e-24f5-4953-aad4-1027fe5d1f95\" (UID: \"d25d107e-24f5-4953-aad4-1027fe5d1f95\") " Jul 15 20:19:28.199697 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:28.199670 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25d107e-24f5-4953-aad4-1027fe5d1f95-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d25d107e-24f5-4953-aad4-1027fe5d1f95" (UID: "d25d107e-24f5-4953-aad4-1027fe5d1f95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:19:28.300464 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:28.300355 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25d107e-24f5-4953-aad4-1027fe5d1f95-maas-api-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:19:28.934804 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:28.934739 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:19:29.023650 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:29.023618 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c923f1-4sv95" event={"ID":"d25d107e-24f5-4953-aad4-1027fe5d1f95","Type":"ContainerDied","Data":"3825a6c44e056ec7d0f012f3e5698e0da5b4f2411252fd35675cf14b70577c93"} Jul 15 20:19:29.023650 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:29.023647 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c923f1-4sv95" Jul 15 20:19:29.023878 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:29.023665 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3825a6c44e056ec7d0f012f3e5698e0da5b4f2411252fd35675cf14b70577c93" Jul 15 20:19:35.702653 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:35.702617 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts"] Jul 15 20:19:35.703308 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:35.702911 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" podUID="e73723d2-c98d-4da5-951d-f0b8bdab6a5f" containerName="maas-api" containerID="cri-o://73d94967a608b7da1d9d62690bb049cc236d22737658c8d138fbe96bb63a60cf" gracePeriod=30 Jul 15 20:19:35.732722 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:35.732690 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:19:35.960231 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:35.960166 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" Jul 15 20:19:36.065399 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:36.065369 2554 generic.go:356] "Generic (PLEG): container finished" podID="e73723d2-c98d-4da5-951d-f0b8bdab6a5f" containerID="73d94967a608b7da1d9d62690bb049cc236d22737658c8d138fbe96bb63a60cf" exitCode=0 Jul 15 20:19:36.065576 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:36.065415 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" event={"ID":"e73723d2-c98d-4da5-951d-f0b8bdab6a5f","Type":"ContainerDied","Data":"73d94967a608b7da1d9d62690bb049cc236d22737658c8d138fbe96bb63a60cf"} Jul 15 20:19:36.065576 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:36.065453 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" event={"ID":"e73723d2-c98d-4da5-951d-f0b8bdab6a5f","Type":"ContainerDied","Data":"cafdf4f380d8440b96ffd844b1dc3a51e7e4bdfa78f47f9e87a0aa5b32279962"} Jul 15 20:19:36.065576 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:36.065473 2554 scope.go:117] "RemoveContainer" containerID="73d94967a608b7da1d9d62690bb049cc236d22737658c8d138fbe96bb63a60cf" Jul 15 20:19:36.065576 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:36.065418 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts" Jul 15 20:19:36.070367 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:36.070341 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqcf8\" (UniqueName: \"kubernetes.io/projected/e73723d2-c98d-4da5-951d-f0b8bdab6a5f-kube-api-access-zqcf8\") pod \"e73723d2-c98d-4da5-951d-f0b8bdab6a5f\" (UID: \"e73723d2-c98d-4da5-951d-f0b8bdab6a5f\") " Jul 15 20:19:36.070489 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:36.070396 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73723d2-c98d-4da5-951d-f0b8bdab6a5f-openshift-service-ca\") pod \"e73723d2-c98d-4da5-951d-f0b8bdab6a5f\" (UID: \"e73723d2-c98d-4da5-951d-f0b8bdab6a5f\") " Jul 15 20:19:36.070489 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:36.070435 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e73723d2-c98d-4da5-951d-f0b8bdab6a5f-maas-api-tls\") pod \"e73723d2-c98d-4da5-951d-f0b8bdab6a5f\" (UID: \"e73723d2-c98d-4da5-951d-f0b8bdab6a5f\") " Jul 15 20:19:36.070791 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:36.070731 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e73723d2-c98d-4da5-951d-f0b8bdab6a5f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e73723d2-c98d-4da5-951d-f0b8bdab6a5f" (UID: "e73723d2-c98d-4da5-951d-f0b8bdab6a5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:19:36.072521 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:36.072498 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73723d2-c98d-4da5-951d-f0b8bdab6a5f-kube-api-access-zqcf8" (OuterVolumeSpecName: "kube-api-access-zqcf8") pod "e73723d2-c98d-4da5-951d-f0b8bdab6a5f" (UID: "e73723d2-c98d-4da5-951d-f0b8bdab6a5f"). InnerVolumeSpecName "kube-api-access-zqcf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:19:36.072602 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:36.072563 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e73723d2-c98d-4da5-951d-f0b8bdab6a5f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e73723d2-c98d-4da5-951d-f0b8bdab6a5f" (UID: "e73723d2-c98d-4da5-951d-f0b8bdab6a5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:19:36.074717 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:36.074697 2554 scope.go:117] "RemoveContainer" containerID="73d94967a608b7da1d9d62690bb049cc236d22737658c8d138fbe96bb63a60cf" Jul 15 20:19:36.075015 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:19:36.074995 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d94967a608b7da1d9d62690bb049cc236d22737658c8d138fbe96bb63a60cf\": container with ID starting with 73d94967a608b7da1d9d62690bb049cc236d22737658c8d138fbe96bb63a60cf not found: ID does not exist" containerID="73d94967a608b7da1d9d62690bb049cc236d22737658c8d138fbe96bb63a60cf" Jul 15 20:19:36.075071 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:36.075021 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d94967a608b7da1d9d62690bb049cc236d22737658c8d138fbe96bb63a60cf"} err="failed to get container status \"73d94967a608b7da1d9d62690bb049cc236d22737658c8d138fbe96bb63a60cf\": rpc error: code = NotFound desc = could not find container \"73d94967a608b7da1d9d62690bb049cc236d22737658c8d138fbe96bb63a60cf\": container with ID starting with 73d94967a608b7da1d9d62690bb049cc236d22737658c8d138fbe96bb63a60cf not found: ID does not exist" Jul 15 20:19:36.171043 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:36.171011 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e73723d2-c98d-4da5-951d-f0b8bdab6a5f-openshift-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:19:36.171043 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:36.171035 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e73723d2-c98d-4da5-951d-f0b8bdab6a5f-maas-api-tls\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:19:36.171043 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:36.171044 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqcf8\" (UniqueName: \"kubernetes.io/projected/e73723d2-c98d-4da5-951d-f0b8bdab6a5f-kube-api-access-zqcf8\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:19:36.384363 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:36.384326 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts"] Jul 15 20:19:36.390337 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:36.390307 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c923f1-799d6d4f4b-tgxts"] Jul 15 20:19:38.239142 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:38.239110 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e73723d2-c98d-4da5-951d-f0b8bdab6a5f" path="/var/lib/kubelet/pods/e73723d2-c98d-4da5-951d-f0b8bdab6a5f/volumes" Jul 15 20:19:40.344413 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:40.344364 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:19:41.834671 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:41.834631 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:19:46.270214 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:46.270167 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:19:46.372372 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:46.372338 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:19:51.047873 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:51.047838 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p"] Jul 15 20:19:51.048295 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:51.048044 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" podUID="7e5742ff-d4c8-4116-b01f-60883b57d9df" containerName="istio-proxy" containerID="cri-o://1d94b6eb62db21f2ba8473a7011cfcfd806576abedb8a86d51fe884a84fd6dbf" gracePeriod=30 Jul 15 20:19:51.072227 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:51.072194 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:19:51.293899 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:51.293856 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:19:51.422571 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:51.422534 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a27bd5-bmttd"] Jul 15 20:19:51.423013 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:51.422996 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d25d107e-24f5-4953-aad4-1027fe5d1f95" containerName="revoke-keys" Jul 15 20:19:51.423070 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:51.423017 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d25d107e-24f5-4953-aad4-1027fe5d1f95" containerName="revoke-keys" Jul 15 20:19:51.423070 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:51.423046 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e73723d2-c98d-4da5-951d-f0b8bdab6a5f" containerName="maas-api" Jul 15 20:19:51.423070 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:51.423055 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e73723d2-c98d-4da5-951d-f0b8bdab6a5f" containerName="maas-api" Jul 15 20:19:51.423188 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:51.423136 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e73723d2-c98d-4da5-951d-f0b8bdab6a5f" containerName="maas-api" Jul 15 20:19:51.423188 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:51.423147 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d25d107e-24f5-4953-aad4-1027fe5d1f95" containerName="revoke-keys" Jul 15 20:19:51.426478 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:51.426460 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a27bd5-bmttd" Jul 15 20:19:51.428977 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:51.428959 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tkj8f\"" Jul 15 20:19:51.435138 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:51.435116 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a27bd5-bmttd"] Jul 15 20:19:51.499505 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:51.499468 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2efc45e0-e8fb-489f-91f1-f3c2ed055f03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a27bd5-bmttd\" (UID: \"2efc45e0-e8fb-489f-91f1-f3c2ed055f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a27bd5-bmttd" Jul 15 20:19:51.600127 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:51.600095 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2efc45e0-e8fb-489f-91f1-f3c2ed055f03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a27bd5-bmttd\" (UID: \"2efc45e0-e8fb-489f-91f1-f3c2ed055f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a27bd5-bmttd" Jul 15 20:19:51.600835 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:51.600790 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2efc45e0-e8fb-489f-91f1-f3c2ed055f03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a27bd5-bmttd\" (UID: \"2efc45e0-e8fb-489f-91f1-f3c2ed055f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a27bd5-bmttd" Jul 15 20:19:51.737677 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:51.737595 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a27bd5-bmttd" Jul 15 20:19:51.883219 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:51.883187 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a27bd5-bmttd"] Jul 15 20:19:51.884847 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:19:51.884803 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2efc45e0_e8fb_489f_91f1_f3c2ed055f03.slice/crio-452a455d63bc81175833f3e5c014d129ef1451158ba1cfe7635575c3b54aacb4 WatchSource:0}: Error finding container 452a455d63bc81175833f3e5c014d129ef1451158ba1cfe7635575c3b54aacb4: Status 404 returned error can't find the container with id 452a455d63bc81175833f3e5c014d129ef1451158ba1cfe7635575c3b54aacb4 Jul 15 20:19:52.065149 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:52.065114 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:19:52.158822 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:52.158780 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:19:52.162458 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:52.162431 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a27bd5-bmttd" event={"ID":"2efc45e0-e8fb-489f-91f1-f3c2ed055f03","Type":"ContainerStarted","Data":"08d3ef9586639c07e33bee2d57ad392f2f3dbb2d207ac7b74549189de6c4fb6c"} Jul 15 20:19:52.162607 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:52.162484 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a27bd5-bmttd" event={"ID":"2efc45e0-e8fb-489f-91f1-f3c2ed055f03","Type":"ContainerStarted","Data":"452a455d63bc81175833f3e5c014d129ef1451158ba1cfe7635575c3b54aacb4"} Jul 15 20:19:52.180936 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:52.180888 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a27bd5-bmttd" podStartSLOduration=1.180872691 podStartE2EDuration="1.180872691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:19:52.179197157 +0000 UTC m=+2506.423703618" watchObservedRunningTime="2026-07-15 20:19:52.180872691 +0000 UTC m=+2506.425379151" Jul 15 20:19:53.169335 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:53.169301 2554 generic.go:356] "Generic (PLEG): container finished" podID="2efc45e0-e8fb-489f-91f1-f3c2ed055f03" containerID="08d3ef9586639c07e33bee2d57ad392f2f3dbb2d207ac7b74549189de6c4fb6c" exitCode=0 Jul 15 20:19:53.169769 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:53.169391 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a27bd5-bmttd" event={"ID":"2efc45e0-e8fb-489f-91f1-f3c2ed055f03","Type":"ContainerDied","Data":"08d3ef9586639c07e33bee2d57ad392f2f3dbb2d207ac7b74549189de6c4fb6c"} Jul 15 20:19:54.311139 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:54.311114 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a27bd5-bmttd" Jul 15 20:19:54.423806 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:54.423731 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2efc45e0-e8fb-489f-91f1-f3c2ed055f03-maas-api-service-ca\") pod \"2efc45e0-e8fb-489f-91f1-f3c2ed055f03\" (UID: \"2efc45e0-e8fb-489f-91f1-f3c2ed055f03\") " Jul 15 20:19:54.424147 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:54.424119 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2efc45e0-e8fb-489f-91f1-f3c2ed055f03-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2efc45e0-e8fb-489f-91f1-f3c2ed055f03" (UID: "2efc45e0-e8fb-489f-91f1-f3c2ed055f03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:19:54.524643 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:54.524538 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2efc45e0-e8fb-489f-91f1-f3c2ed055f03-maas-api-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:19:55.182418 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:55.182385 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a27bd5-bmttd" event={"ID":"2efc45e0-e8fb-489f-91f1-f3c2ed055f03","Type":"ContainerDied","Data":"452a455d63bc81175833f3e5c014d129ef1451158ba1cfe7635575c3b54aacb4"} Jul 15 20:19:55.182418 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:55.182415 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a27bd5-bmttd" Jul 15 20:19:55.182644 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:55.182433 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="452a455d63bc81175833f3e5c014d129ef1451158ba1cfe7635575c3b54aacb4" Jul 15 20:19:56.191596 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.191564 2554 generic.go:356] "Generic (PLEG): container finished" podID="7e5742ff-d4c8-4116-b01f-60883b57d9df" containerID="1d94b6eb62db21f2ba8473a7011cfcfd806576abedb8a86d51fe884a84fd6dbf" exitCode=0 Jul 15 20:19:56.192069 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.191645 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" event={"ID":"7e5742ff-d4c8-4116-b01f-60883b57d9df","Type":"ContainerDied","Data":"1d94b6eb62db21f2ba8473a7011cfcfd806576abedb8a86d51fe884a84fd6dbf"} Jul 15 20:19:56.192069 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.191723 2554 scope.go:117] "RemoveContainer" containerID="3bb8068676edc54483a4e253355e97e7b92937c7c85d3c4a000c1235d40f334e" Jul 15 20:19:56.307571 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.307500 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:19:56.443062 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.443027 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-envoy\") pod \"7e5742ff-d4c8-4116-b01f-60883b57d9df\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " Jul 15 20:19:56.443260 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.443076 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5742ff-d4c8-4116-b01f-60883b57d9df-istiod-ca-cert\") pod \"7e5742ff-d4c8-4116-b01f-60883b57d9df\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " Jul 15 20:19:56.443260 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.443097 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-workload-socket\") pod \"7e5742ff-d4c8-4116-b01f-60883b57d9df\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " Jul 15 20:19:56.443260 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.443137 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-podinfo\") pod \"7e5742ff-d4c8-4116-b01f-60883b57d9df\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " Jul 15 20:19:56.443260 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.443180 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-token\") pod \"7e5742ff-d4c8-4116-b01f-60883b57d9df\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " Jul 15 20:19:56.443260 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.443216 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-workload-certs\") pod \"7e5742ff-d4c8-4116-b01f-60883b57d9df\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " Jul 15 20:19:56.443260 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.443245 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-452dc\" (UniqueName: \"kubernetes.io/projected/7e5742ff-d4c8-4116-b01f-60883b57d9df-kube-api-access-452dc\") pod \"7e5742ff-d4c8-4116-b01f-60883b57d9df\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " Jul 15 20:19:56.443577 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.443285 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-credential-socket\") pod \"7e5742ff-d4c8-4116-b01f-60883b57d9df\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " Jul 15 20:19:56.443577 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.443305 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-data\") pod \"7e5742ff-d4c8-4116-b01f-60883b57d9df\" (UID: \"7e5742ff-d4c8-4116-b01f-60883b57d9df\") " Jul 15 20:19:56.443577 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.443433 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7e5742ff-d4c8-4116-b01f-60883b57d9df" (UID: "7e5742ff-d4c8-4116-b01f-60883b57d9df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:19:56.443717 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.443600 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-workload-socket\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:19:56.443717 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.443621 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5742ff-d4c8-4116-b01f-60883b57d9df-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7e5742ff-d4c8-4116-b01f-60883b57d9df" (UID: "7e5742ff-d4c8-4116-b01f-60883b57d9df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:19:56.443857 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.443829 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7e5742ff-d4c8-4116-b01f-60883b57d9df" (UID: "7e5742ff-d4c8-4116-b01f-60883b57d9df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:19:56.443916 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.443857 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7e5742ff-d4c8-4116-b01f-60883b57d9df" (UID: "7e5742ff-d4c8-4116-b01f-60883b57d9df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:19:56.444513 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.444483 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-data" (OuterVolumeSpecName: "istio-data") pod "7e5742ff-d4c8-4116-b01f-60883b57d9df" (UID: "7e5742ff-d4c8-4116-b01f-60883b57d9df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:19:56.445617 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.445581 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7e5742ff-d4c8-4116-b01f-60883b57d9df" (UID: "7e5742ff-d4c8-4116-b01f-60883b57d9df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:19:56.445617 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.445594 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7e5742ff-d4c8-4116-b01f-60883b57d9df" (UID: "7e5742ff-d4c8-4116-b01f-60883b57d9df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:19:56.445807 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.445637 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5742ff-d4c8-4116-b01f-60883b57d9df-kube-api-access-452dc" (OuterVolumeSpecName: "kube-api-access-452dc") pod "7e5742ff-d4c8-4116-b01f-60883b57d9df" (UID: "7e5742ff-d4c8-4116-b01f-60883b57d9df"). InnerVolumeSpecName "kube-api-access-452dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:19:56.445807 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.445644 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e5742ff-d4c8-4116-b01f-60883b57d9df" (UID: "7e5742ff-d4c8-4116-b01f-60883b57d9df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:19:56.544804 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.544738 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-podinfo\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:19:56.544804 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.544799 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-token\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:19:56.544804 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.544817 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-workload-certs\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:19:56.545065 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.544826 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-452dc\" (UniqueName: \"kubernetes.io/projected/7e5742ff-d4c8-4116-b01f-60883b57d9df-kube-api-access-452dc\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:19:56.545065 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.544835 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-credential-socket\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:19:56.545065 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.544843 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-data\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:19:56.545065 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.544850 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5742ff-d4c8-4116-b01f-60883b57d9df-istio-envoy\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:19:56.545065 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:56.544858 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5742ff-d4c8-4116-b01f-60883b57d9df-istiod-ca-cert\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:19:57.198927 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:57.198852 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" Jul 15 20:19:57.198927 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:57.198858 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p" event={"ID":"7e5742ff-d4c8-4116-b01f-60883b57d9df","Type":"ContainerDied","Data":"7466fc84288ea3c6b429cd4183ca14d9fb10a48cafa5156b77acdb1614e903f7"} Jul 15 20:19:57.198927 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:57.198914 2554 scope.go:117] "RemoveContainer" containerID="1d94b6eb62db21f2ba8473a7011cfcfd806576abedb8a86d51fe884a84fd6dbf" Jul 15 20:19:57.229476 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:57.229438 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p"] Jul 15 20:19:57.236614 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:57.236587 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c923f1-openshift-default-5c8d64f885-bb48p"] Jul 15 20:19:58.239582 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:19:58.239545 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e5742ff-d4c8-4116-b01f-60883b57d9df" path="/var/lib/kubelet/pods/7e5742ff-d4c8-4116-b01f-60883b57d9df/volumes" Jul 15 20:20:02.032038 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:02.032006 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf"] Jul 15 20:20:02.032396 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:02.032229 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" podUID="a6a6549a-28c8-4670-ab2e-ff04e5a746cc" containerName="maas-api" containerID="cri-o://19dc775d7d749352a812609c26f1f5689cbb3a961dad051e3b4fa9fea8fbb3bd" gracePeriod=30 Jul 15 20:20:02.071570 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:02.071541 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:02.235407 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:02.235372 2554 generic.go:356] "Generic (PLEG): container finished" podID="a6a6549a-28c8-4670-ab2e-ff04e5a746cc" containerID="19dc775d7d749352a812609c26f1f5689cbb3a961dad051e3b4fa9fea8fbb3bd" exitCode=0 Jul 15 20:20:02.239909 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:02.239878 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" event={"ID":"a6a6549a-28c8-4670-ab2e-ff04e5a746cc","Type":"ContainerDied","Data":"19dc775d7d749352a812609c26f1f5689cbb3a961dad051e3b4fa9fea8fbb3bd"} Jul 15 20:20:02.288894 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:02.288831 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" Jul 15 20:20:02.393217 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:02.393183 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a6549a-28c8-4670-ab2e-ff04e5a746cc-maas-api-tls\") pod \"a6a6549a-28c8-4670-ab2e-ff04e5a746cc\" (UID: \"a6a6549a-28c8-4670-ab2e-ff04e5a746cc\") " Jul 15 20:20:02.393395 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:02.393248 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a6549a-28c8-4670-ab2e-ff04e5a746cc-openshift-service-ca\") pod \"a6a6549a-28c8-4670-ab2e-ff04e5a746cc\" (UID: \"a6a6549a-28c8-4670-ab2e-ff04e5a746cc\") " Jul 15 20:20:02.393395 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:02.393279 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmn57\" (UniqueName: \"kubernetes.io/projected/a6a6549a-28c8-4670-ab2e-ff04e5a746cc-kube-api-access-mmn57\") pod \"a6a6549a-28c8-4670-ab2e-ff04e5a746cc\" (UID: \"a6a6549a-28c8-4670-ab2e-ff04e5a746cc\") " Jul 15 20:20:02.393582 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:02.393557 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a6549a-28c8-4670-ab2e-ff04e5a746cc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a6a6549a-28c8-4670-ab2e-ff04e5a746cc" (UID: "a6a6549a-28c8-4670-ab2e-ff04e5a746cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:20:02.395385 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:02.395352 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6a6549a-28c8-4670-ab2e-ff04e5a746cc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a6a6549a-28c8-4670-ab2e-ff04e5a746cc" (UID: "a6a6549a-28c8-4670-ab2e-ff04e5a746cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:20:02.395485 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:02.395458 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a6549a-28c8-4670-ab2e-ff04e5a746cc-kube-api-access-mmn57" (OuterVolumeSpecName: "kube-api-access-mmn57") pod "a6a6549a-28c8-4670-ab2e-ff04e5a746cc" (UID: "a6a6549a-28c8-4670-ab2e-ff04e5a746cc"). InnerVolumeSpecName "kube-api-access-mmn57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:20:02.494108 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:02.494072 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a6549a-28c8-4670-ab2e-ff04e5a746cc-maas-api-tls\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:20:02.494108 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:02.494102 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a6549a-28c8-4670-ab2e-ff04e5a746cc-openshift-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:20:02.494108 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:02.494112 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmn57\" (UniqueName: \"kubernetes.io/projected/a6a6549a-28c8-4670-ab2e-ff04e5a746cc-kube-api-access-mmn57\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:20:03.242265 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:03.242222 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" event={"ID":"a6a6549a-28c8-4670-ab2e-ff04e5a746cc","Type":"ContainerDied","Data":"b985ca4b5dbf91ab9d1d686795046636f86cfa7c93bd55073af140e9c102029c"} Jul 15 20:20:03.242661 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:03.242291 2554 scope.go:117] "RemoveContainer" containerID="19dc775d7d749352a812609c26f1f5689cbb3a961dad051e3b4fa9fea8fbb3bd" Jul 15 20:20:03.242661 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:03.242249 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf" Jul 15 20:20:03.265081 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:03.265051 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf"] Jul 15 20:20:03.269476 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:03.269452 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a27bd5-595dfdfc66-5dzmf"] Jul 15 20:20:04.239127 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:04.239090 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6a6549a-28c8-4670-ab2e-ff04e5a746cc" path="/var/lib/kubelet/pods/a6a6549a-28c8-4670-ab2e-ff04e5a746cc/volumes" Jul 15 20:20:05.104294 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:05.104259 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:20:05.259822 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:05.259789 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:06.531300 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:06.531259 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:12.447578 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:12.447541 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:13.066709 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:13.066678 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:13.143739 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:13.143693 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:17.274477 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:17.274417 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:17.468024 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:17.467986 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:17.918100 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:17.918051 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:18.107765 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.107717 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9"] Jul 15 20:20:18.108064 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.108052 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5742ff-d4c8-4116-b01f-60883b57d9df" containerName="istio-proxy" Jul 15 20:20:18.108110 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.108065 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5742ff-d4c8-4116-b01f-60883b57d9df" containerName="istio-proxy" Jul 15 20:20:18.108110 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.108086 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6a6549a-28c8-4670-ab2e-ff04e5a746cc" containerName="maas-api" Jul 15 20:20:18.108110 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.108091 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6a6549a-28c8-4670-ab2e-ff04e5a746cc" containerName="maas-api" Jul 15 20:20:18.108110 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.108097 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2efc45e0-e8fb-489f-91f1-f3c2ed055f03" containerName="revoke-keys" Jul 15 20:20:18.108110 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.108102 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2efc45e0-e8fb-489f-91f1-f3c2ed055f03" containerName="revoke-keys" Jul 15 20:20:18.108272 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.108157 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6a6549a-28c8-4670-ab2e-ff04e5a746cc" containerName="maas-api" Jul 15 20:20:18.108272 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.108168 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e5742ff-d4c8-4116-b01f-60883b57d9df" containerName="istio-proxy" Jul 15 20:20:18.108272 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.108174 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2efc45e0-e8fb-489f-91f1-f3c2ed055f03" containerName="revoke-keys" Jul 15 20:20:18.111137 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.111120 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.113776 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.113728 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cae23a-openshift-default-dockercfg-bkr7w\"" Jul 15 20:20:18.124303 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.124273 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9"] Jul 15 20:20:18.126609 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.126579 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-workload-socket\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.126770 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.126652 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db02b3f4-8996-4918-acfe-818617c4866a-istio-token\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.126770 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.126690 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db02b3f4-8996-4918-acfe-818617c4866a-istio-podinfo\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.126922 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.126788 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjvpt\" (UniqueName: \"kubernetes.io/projected/db02b3f4-8996-4918-acfe-818617c4866a-kube-api-access-wjvpt\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.126922 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.126813 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-istio-envoy\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.126922 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.126839 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-credential-socket\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.126922 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.126873 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-istio-data\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.127129 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.126963 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-workload-certs\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.127129 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.126999 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db02b3f4-8996-4918-acfe-818617c4866a-istiod-ca-cert\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.127366 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.127343 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:18.227528 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.227429 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-workload-socket\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.227528 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.227465 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db02b3f4-8996-4918-acfe-818617c4866a-istio-token\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.227528 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.227486 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db02b3f4-8996-4918-acfe-818617c4866a-istio-podinfo\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.227528 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.227518 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjvpt\" (UniqueName: \"kubernetes.io/projected/db02b3f4-8996-4918-acfe-818617c4866a-kube-api-access-wjvpt\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.228023 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.227542 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-istio-envoy\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.228023 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.227575 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-credential-socket\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.228023 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.227606 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-istio-data\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.228023 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.227643 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-workload-certs\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.228023 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.227674 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db02b3f4-8996-4918-acfe-818617c4866a-istiod-ca-cert\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.228023 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.227937 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-workload-socket\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.228497 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.228111 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-workload-certs\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.228497 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.228187 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-credential-socket\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.228497 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.228363 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-istio-data\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.228497 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.228415 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db02b3f4-8996-4918-acfe-818617c4866a-istiod-ca-cert\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.230052 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.230028 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-istio-envoy\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.230324 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.230305 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db02b3f4-8996-4918-acfe-818617c4866a-istio-podinfo\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.236360 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.236329 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db02b3f4-8996-4918-acfe-818617c4866a-istio-token\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.236360 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.236353 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjvpt\" (UniqueName: \"kubernetes.io/projected/db02b3f4-8996-4918-acfe-818617c4866a-kube-api-access-wjvpt\") pod \"e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.422323 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.422281 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:18.479576 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.479488 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:18.570839 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.570197 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9"] Jul 15 20:20:18.573234 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:20:18.573204 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb02b3f4_8996_4918_acfe_818617c4866a.slice/crio-d97562cad3185110a666fbb951a337eee2d4d33063666930406ccd472d2a4765 WatchSource:0}: Error finding container d97562cad3185110a666fbb951a337eee2d4d33063666930406ccd472d2a4765: Status 404 returned error can't find the container with id d97562cad3185110a666fbb951a337eee2d4d33063666930406ccd472d2a4765 Jul 15 20:20:18.575741 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.575702 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:20:18.575839 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.575821 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:20:18.575881 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:18.575867 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:20:19.332719 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:19.332682 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" event={"ID":"db02b3f4-8996-4918-acfe-818617c4866a","Type":"ContainerStarted","Data":"607a7368bc32f25efeaaea71d2cf530305d2a78e81804a496337dafeb8e368b6"} Jul 15 20:20:19.332960 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:19.332735 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" event={"ID":"db02b3f4-8996-4918-acfe-818617c4866a","Type":"ContainerStarted","Data":"d97562cad3185110a666fbb951a337eee2d4d33063666930406ccd472d2a4765"} Jul 15 20:20:19.355643 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:19.355590 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" podStartSLOduration=1.355573653 podStartE2EDuration="1.355573653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:20:19.353418517 +0000 UTC m=+2533.597924978" watchObservedRunningTime="2026-07-15 20:20:19.355573653 +0000 UTC m=+2533.600080115" Jul 15 20:20:19.423506 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:19.423460 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:19.428516 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:19.428488 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:20.338813 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:20.338778 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:20.339711 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:20.339693 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:21.376627 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.376584 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns"] Jul 15 20:20:21.377155 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.377135 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5742ff-d4c8-4116-b01f-60883b57d9df" containerName="istio-proxy" Jul 15 20:20:21.377223 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.377158 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5742ff-d4c8-4116-b01f-60883b57d9df" containerName="istio-proxy" Jul 15 20:20:21.377294 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.377281 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e5742ff-d4c8-4116-b01f-60883b57d9df" containerName="istio-proxy" Jul 15 20:20:21.380669 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.380643 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" Jul 15 20:20:21.383648 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.383620 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-cae23a-serving-cert\"" Jul 15 20:20:21.383648 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.383635 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 20:20:21.383844 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.383665 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 20:20:21.384646 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.384631 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c5cgj\"" Jul 15 20:20:21.396155 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.396123 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns"] Jul 15 20:20:21.454573 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.454525 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a96b977-6bc2-4826-b541-21251659114b-maas-api-tls\") pod \"maas-api-e2e-api-a-cae23a-6c696fc4-jkfns\" (UID: \"4a96b977-6bc2-4826-b541-21251659114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" Jul 15 20:20:21.454573 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.454581 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjhph\" (UniqueName: \"kubernetes.io/projected/4a96b977-6bc2-4826-b541-21251659114b-kube-api-access-sjhph\") pod \"maas-api-e2e-api-a-cae23a-6c696fc4-jkfns\" (UID: \"4a96b977-6bc2-4826-b541-21251659114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" Jul 15 20:20:21.454831 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.454603 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a96b977-6bc2-4826-b541-21251659114b-openshift-service-ca\") pod \"maas-api-e2e-api-a-cae23a-6c696fc4-jkfns\" (UID: \"4a96b977-6bc2-4826-b541-21251659114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" Jul 15 20:20:21.555174 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.555139 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjhph\" (UniqueName: \"kubernetes.io/projected/4a96b977-6bc2-4826-b541-21251659114b-kube-api-access-sjhph\") pod \"maas-api-e2e-api-a-cae23a-6c696fc4-jkfns\" (UID: \"4a96b977-6bc2-4826-b541-21251659114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" Jul 15 20:20:21.555174 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.555179 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a96b977-6bc2-4826-b541-21251659114b-openshift-service-ca\") pod \"maas-api-e2e-api-a-cae23a-6c696fc4-jkfns\" (UID: \"4a96b977-6bc2-4826-b541-21251659114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" Jul 15 20:20:21.555478 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.555285 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a96b977-6bc2-4826-b541-21251659114b-maas-api-tls\") pod \"maas-api-e2e-api-a-cae23a-6c696fc4-jkfns\" (UID: \"4a96b977-6bc2-4826-b541-21251659114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" Jul 15 20:20:21.556049 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.556026 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a96b977-6bc2-4826-b541-21251659114b-openshift-service-ca\") pod \"maas-api-e2e-api-a-cae23a-6c696fc4-jkfns\" (UID: \"4a96b977-6bc2-4826-b541-21251659114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" Jul 15 20:20:21.558058 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.558035 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a96b977-6bc2-4826-b541-21251659114b-maas-api-tls\") pod \"maas-api-e2e-api-a-cae23a-6c696fc4-jkfns\" (UID: \"4a96b977-6bc2-4826-b541-21251659114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" Jul 15 20:20:21.575566 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.575524 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjhph\" (UniqueName: \"kubernetes.io/projected/4a96b977-6bc2-4826-b541-21251659114b-kube-api-access-sjhph\") pod \"maas-api-e2e-api-a-cae23a-6c696fc4-jkfns\" (UID: \"4a96b977-6bc2-4826-b541-21251659114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" Jul 15 20:20:21.692213 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.692171 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" Jul 15 20:20:21.835948 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.835917 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns"] Jul 15 20:20:21.836842 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:20:21.836815 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a96b977_6bc2_4826_b541_21251659114b.slice/crio-a59a4be547449deb899b8ecb03b628f29452bbf53cffd126fdcf425a70987f29 WatchSource:0}: Error finding container a59a4be547449deb899b8ecb03b628f29452bbf53cffd126fdcf425a70987f29: Status 404 returned error can't find the container with id a59a4be547449deb899b8ecb03b628f29452bbf53cffd126fdcf425a70987f29 Jul 15 20:20:21.838329 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.838312 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 20:20:21.923070 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:21.923030 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:22.024010 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:22.023919 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:22.125686 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:22.125648 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:22.227388 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:22.227355 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:22.351979 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:22.351942 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" event={"ID":"4a96b977-6bc2-4826-b541-21251659114b","Type":"ContainerStarted","Data":"2b08301a1a7df122ab820a34247afb93784a45be87bf41c660957e5ab2fced42"} Jul 15 20:20:22.352153 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:22.352007 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" event={"ID":"4a96b977-6bc2-4826-b541-21251659114b","Type":"ContainerStarted","Data":"a59a4be547449deb899b8ecb03b628f29452bbf53cffd126fdcf425a70987f29"} Jul 15 20:20:22.352225 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:22.352144 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" Jul 15 20:20:22.383023 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:22.382967 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" podStartSLOduration=0.943758178 podStartE2EDuration="1.382947731s" totalImagesPullingTime="439.189553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:20:22.381953255 +0000 UTC m=+2536.626459717" watchObservedRunningTime="2026-07-15 20:20:22.382947731 +0000 UTC m=+2536.627454192" Jul 15 20:20:25.635833 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:25.635798 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:20:25.822590 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:25.822558 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:27.049332 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:27.049288 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:28.364841 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:28.364806 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" Jul 15 20:20:29.805942 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:29.805909 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:29.890149 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:29.890114 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t"] Jul 15 20:20:29.895084 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:29.895057 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" Jul 15 20:20:29.897782 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:29.897735 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-772fb7-serving-cert\"" Jul 15 20:20:29.908217 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:29.907218 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t"] Jul 15 20:20:29.915062 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:29.915022 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:29.939187 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:29.939148 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pmhl\" (UniqueName: \"kubernetes.io/projected/2d056ca7-4fe4-4787-9716-242dae9ac81b-kube-api-access-9pmhl\") pod \"maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t\" (UID: \"2d056ca7-4fe4-4787-9716-242dae9ac81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" Jul 15 20:20:29.939536 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:29.939511 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d056ca7-4fe4-4787-9716-242dae9ac81b-openshift-service-ca\") pod \"maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t\" (UID: \"2d056ca7-4fe4-4787-9716-242dae9ac81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" Jul 15 20:20:29.939745 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:29.939728 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d056ca7-4fe4-4787-9716-242dae9ac81b-maas-api-tls\") pod \"maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t\" (UID: \"2d056ca7-4fe4-4787-9716-242dae9ac81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" Jul 15 20:20:30.001879 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:30.001846 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:30.040715 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:30.040669 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d056ca7-4fe4-4787-9716-242dae9ac81b-maas-api-tls\") pod \"maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t\" (UID: \"2d056ca7-4fe4-4787-9716-242dae9ac81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" Jul 15 20:20:30.041058 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:30.041032 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pmhl\" (UniqueName: \"kubernetes.io/projected/2d056ca7-4fe4-4787-9716-242dae9ac81b-kube-api-access-9pmhl\") pod \"maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t\" (UID: \"2d056ca7-4fe4-4787-9716-242dae9ac81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" Jul 15 20:20:30.041239 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:30.041224 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d056ca7-4fe4-4787-9716-242dae9ac81b-openshift-service-ca\") pod \"maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t\" (UID: \"2d056ca7-4fe4-4787-9716-242dae9ac81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" Jul 15 20:20:30.042142 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:30.042117 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d056ca7-4fe4-4787-9716-242dae9ac81b-openshift-service-ca\") pod \"maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t\" (UID: \"2d056ca7-4fe4-4787-9716-242dae9ac81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" Jul 15 20:20:30.042291 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:20:30.040915 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-serving-cert: secret "maas-api-e2e-api-b-772fb7-serving-cert" not found Jul 15 20:20:30.042434 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:20:30.042422 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d056ca7-4fe4-4787-9716-242dae9ac81b-maas-api-tls podName:2d056ca7-4fe4-4787-9716-242dae9ac81b nodeName:}" failed. No retries permitted until 2026-07-15 20:20:30.542399877 +0000 UTC m=+2544.786906322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d056ca7-4fe4-4787-9716-242dae9ac81b-maas-api-tls") pod "maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" (UID: "2d056ca7-4fe4-4787-9716-242dae9ac81b") : secret "maas-api-e2e-api-b-772fb7-serving-cert" not found Jul 15 20:20:30.054732 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:30.054699 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pmhl\" (UniqueName: \"kubernetes.io/projected/2d056ca7-4fe4-4787-9716-242dae9ac81b-kube-api-access-9pmhl\") pod \"maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t\" (UID: \"2d056ca7-4fe4-4787-9716-242dae9ac81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" Jul 15 20:20:30.465887 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:30.465856 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:30.553649 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:30.553603 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d056ca7-4fe4-4787-9716-242dae9ac81b-maas-api-tls\") pod \"maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t\" (UID: \"2d056ca7-4fe4-4787-9716-242dae9ac81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" Jul 15 20:20:30.556313 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:30.556289 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d056ca7-4fe4-4787-9716-242dae9ac81b-maas-api-tls\") pod \"maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t\" (UID: \"2d056ca7-4fe4-4787-9716-242dae9ac81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" Jul 15 20:20:30.811650 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:30.811551 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" Jul 15 20:20:30.959779 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:30.959736 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t"] Jul 15 20:20:30.961805 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:20:30.961771 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d056ca7_4fe4_4787_9716_242dae9ac81b.slice/crio-45afc4186d34774da01023f3e2c614793417468233c12056412dbaf670898e97 WatchSource:0}: Error finding container 45afc4186d34774da01023f3e2c614793417468233c12056412dbaf670898e97: Status 404 returned error can't find the container with id 45afc4186d34774da01023f3e2c614793417468233c12056412dbaf670898e97 Jul 15 20:20:31.360089 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:31.360060 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:31.407981 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:31.407939 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" event={"ID":"2d056ca7-4fe4-4787-9716-242dae9ac81b","Type":"ContainerStarted","Data":"45afc4186d34774da01023f3e2c614793417468233c12056412dbaf670898e97"} Jul 15 20:20:31.463126 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:31.463097 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:32.415865 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:32.415822 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" event={"ID":"2d056ca7-4fe4-4787-9716-242dae9ac81b","Type":"ContainerStarted","Data":"e2ab0e864a79dbf9f59cd3b4b97ff83eb07f11ece3e247dc74bbaac0c837bcac"} Jul 15 20:20:32.416260 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:32.415890 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" Jul 15 20:20:32.436497 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:32.436430 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" podStartSLOduration=3.032016135 podStartE2EDuration="3.436409935s" totalImagesPullingTime="404.3938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:20:32.432012128 +0000 UTC m=+2546.676518597" watchObservedRunningTime="2026-07-15 20:20:32.436409935 +0000 UTC m=+2546.680916397" Jul 15 20:20:34.326953 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:34.326922 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:34.427730 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:34.427695 2554 generic.go:356] "Generic (PLEG): container finished" podID="db02b3f4-8996-4918-acfe-818617c4866a" containerID="607a7368bc32f25efeaaea71d2cf530305d2a78e81804a496337dafeb8e368b6" exitCode=0 Jul 15 20:20:34.427906 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:34.427789 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" event={"ID":"db02b3f4-8996-4918-acfe-818617c4866a","Type":"ContainerDied","Data":"607a7368bc32f25efeaaea71d2cf530305d2a78e81804a496337dafeb8e368b6"} Jul 15 20:20:34.428190 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:34.428177 2554 scope.go:117] "RemoveContainer" containerID="607a7368bc32f25efeaaea71d2cf530305d2a78e81804a496337dafeb8e368b6" Jul 15 20:20:34.429185 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:34.429158 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:20:34.429239 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:34.429224 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:20:34.429274 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:34.429254 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:20:35.339471 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:35.339429 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:35.440584 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:35.440535 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" event={"ID":"db02b3f4-8996-4918-acfe-818617c4866a","Type":"ContainerStarted","Data":"03ef748925382288a8a9ba4a09d19b4e05d9aa7612b77f967b56ed743ac67970"} Jul 15 20:20:36.422439 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:36.422396 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:36.423995 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:36.423957 2554 patch_prober.go:28] interesting pod/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 15 20:20:36.424142 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:36.424025 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" podUID="db02b3f4-8996-4918-acfe-818617c4866a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 15 20:20:37.422888 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:37.422849 2554 patch_prober.go:28] interesting pod/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 15 20:20:37.423498 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:37.423463 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" podUID="db02b3f4-8996-4918-acfe-818617c4866a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 15 20:20:37.436580 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:37.436544 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:20:38.423995 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:38.423956 2554 patch_prober.go:28] interesting pod/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 15 20:20:38.424574 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:38.424542 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" podUID="db02b3f4-8996-4918-acfe-818617c4866a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 15 20:20:38.434975 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:38.434950 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" Jul 15 20:20:39.426930 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:39.426900 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:39.427319 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:39.427265 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:39.427931 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:39.427912 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:20:42.842794 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:42.842738 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:42.940716 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:42.940682 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:47.341427 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:47.341374 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:47.399508 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:47.399455 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:51.463228 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:51.463183 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:51.560969 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:51.560934 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:53.170365 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:53.170332 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:53.412075 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:53.412041 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:53.413252 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:53.413228 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns"] Jul 15 20:20:53.413489 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:53.413460 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" podUID="4a96b977-6bc2-4826-b541-21251659114b" containerName="maas-api" containerID="cri-o://2b08301a1a7df122ab820a34247afb93784a45be87bf41c660957e5ab2fced42" gracePeriod=30 Jul 15 20:20:53.510231 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:53.510201 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:20:53.550553 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:53.550516 2554 generic.go:356] "Generic (PLEG): container finished" podID="4a96b977-6bc2-4826-b541-21251659114b" containerID="2b08301a1a7df122ab820a34247afb93784a45be87bf41c660957e5ab2fced42" exitCode=0 Jul 15 20:20:53.550740 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:53.550600 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" event={"ID":"4a96b977-6bc2-4826-b541-21251659114b","Type":"ContainerDied","Data":"2b08301a1a7df122ab820a34247afb93784a45be87bf41c660957e5ab2fced42"} Jul 15 20:20:53.669766 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:53.669733 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" Jul 15 20:20:53.674516 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:53.674493 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a96b977-6bc2-4826-b541-21251659114b-openshift-service-ca\") pod \"4a96b977-6bc2-4826-b541-21251659114b\" (UID: \"4a96b977-6bc2-4826-b541-21251659114b\") " Jul 15 20:20:53.674616 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:53.674550 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjhph\" (UniqueName: \"kubernetes.io/projected/4a96b977-6bc2-4826-b541-21251659114b-kube-api-access-sjhph\") pod \"4a96b977-6bc2-4826-b541-21251659114b\" (UID: \"4a96b977-6bc2-4826-b541-21251659114b\") " Jul 15 20:20:53.674616 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:53.674573 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a96b977-6bc2-4826-b541-21251659114b-maas-api-tls\") pod \"4a96b977-6bc2-4826-b541-21251659114b\" (UID: \"4a96b977-6bc2-4826-b541-21251659114b\") " Jul 15 20:20:53.674926 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:53.674901 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a96b977-6bc2-4826-b541-21251659114b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a96b977-6bc2-4826-b541-21251659114b" (UID: "4a96b977-6bc2-4826-b541-21251659114b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:20:53.676837 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:53.676777 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a96b977-6bc2-4826-b541-21251659114b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a96b977-6bc2-4826-b541-21251659114b" (UID: "4a96b977-6bc2-4826-b541-21251659114b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:20:53.676917 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:53.676894 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a96b977-6bc2-4826-b541-21251659114b-kube-api-access-sjhph" (OuterVolumeSpecName: "kube-api-access-sjhph") pod "4a96b977-6bc2-4826-b541-21251659114b" (UID: "4a96b977-6bc2-4826-b541-21251659114b"). InnerVolumeSpecName "kube-api-access-sjhph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:20:53.775276 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:53.775232 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjhph\" (UniqueName: \"kubernetes.io/projected/4a96b977-6bc2-4826-b541-21251659114b-kube-api-access-sjhph\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:20:53.775276 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:53.775271 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a96b977-6bc2-4826-b541-21251659114b-maas-api-tls\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:20:53.775276 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:53.775281 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a96b977-6bc2-4826-b541-21251659114b-openshift-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:20:54.558387 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:54.558301 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" Jul 15 20:20:54.558387 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:54.558326 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns" event={"ID":"4a96b977-6bc2-4826-b541-21251659114b","Type":"ContainerDied","Data":"a59a4be547449deb899b8ecb03b628f29452bbf53cffd126fdcf425a70987f29"} Jul 15 20:20:54.558879 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:54.558390 2554 scope.go:117] "RemoveContainer" containerID="2b08301a1a7df122ab820a34247afb93784a45be87bf41c660957e5ab2fced42" Jul 15 20:20:54.576386 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:54.576355 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns"] Jul 15 20:20:54.580826 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:54.580793 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cae23a-6c696fc4-jkfns"] Jul 15 20:20:56.239637 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:56.239597 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a96b977-6bc2-4826-b541-21251659114b" path="/var/lib/kubelet/pods/4a96b977-6bc2-4826-b541-21251659114b/volumes" Jul 15 20:20:56.933466 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:56.933426 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:20:57.556946 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:20:57.556907 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:03.535257 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:03.535218 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:03.637073 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:03.637033 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:06.614872 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:06.614833 2554 scope.go:117] "RemoveContainer" containerID="effc5318df75df1a1fe8a9d3fe5ec4a9f94f2f75de1b41a6e16a23bb597ff2bf" Jul 15 20:21:08.201297 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:08.201261 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9"] Jul 15 20:21:08.201807 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:08.201515 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" podUID="db02b3f4-8996-4918-acfe-818617c4866a" containerName="istio-proxy" containerID="cri-o://03ef748925382288a8a9ba4a09d19b4e05d9aa7612b77f967b56ed743ac67970" gracePeriod=30 Jul 15 20:21:08.216723 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:08.216694 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:08.467989 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:08.467900 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:08.585371 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:08.585328 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-772fb7-mlhr2"] Jul 15 20:21:08.585706 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:08.585692 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a96b977-6bc2-4826-b541-21251659114b" containerName="maas-api" Jul 15 20:21:08.585706 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:08.585708 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a96b977-6bc2-4826-b541-21251659114b" containerName="maas-api" Jul 15 20:21:08.585851 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:08.585798 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a96b977-6bc2-4826-b541-21251659114b" containerName="maas-api" Jul 15 20:21:08.590457 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:08.590434 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-772fb7-mlhr2" Jul 15 20:21:08.593063 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:08.593043 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tkj8f\"" Jul 15 20:21:08.599113 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:08.599086 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-772fb7-mlhr2"] Jul 15 20:21:08.709907 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:08.709863 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d28d4f7e-710c-4409-88c1-ad1610a7d930-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-772fb7-mlhr2\" (UID: \"d28d4f7e-710c-4409-88c1-ad1610a7d930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-772fb7-mlhr2" Jul 15 20:21:08.810544 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:08.810444 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d28d4f7e-710c-4409-88c1-ad1610a7d930-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-772fb7-mlhr2\" (UID: \"d28d4f7e-710c-4409-88c1-ad1610a7d930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-772fb7-mlhr2" Jul 15 20:21:08.811131 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:08.811106 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d28d4f7e-710c-4409-88c1-ad1610a7d930-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-772fb7-mlhr2\" (UID: \"d28d4f7e-710c-4409-88c1-ad1610a7d930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-772fb7-mlhr2" Jul 15 20:21:08.901600 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:08.901561 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-772fb7-mlhr2" Jul 15 20:21:09.042789 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:09.042745 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-772fb7-mlhr2"] Jul 15 20:21:09.044733 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:21:09.044705 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28d4f7e_710c_4409_88c1_ad1610a7d930.slice/crio-6baaa8a628ddb03294789694013511c78239f13b798d3d5eb55faaedbdd7a22c WatchSource:0}: Error finding container 6baaa8a628ddb03294789694013511c78239f13b798d3d5eb55faaedbdd7a22c: Status 404 returned error can't find the container with id 6baaa8a628ddb03294789694013511c78239f13b798d3d5eb55faaedbdd7a22c Jul 15 20:21:09.231364 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:09.231305 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:09.331318 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:09.331226 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:09.429847 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:09.429809 2554 patch_prober.go:28] interesting pod/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 20:21:09.430000 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:09.429870 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" podUID="db02b3f4-8996-4918-acfe-818617c4866a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 20:21:09.652904 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:09.652799 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-772fb7-mlhr2" event={"ID":"d28d4f7e-710c-4409-88c1-ad1610a7d930","Type":"ContainerStarted","Data":"7318210c29f289f0c19e1887a4c339d96fed56cba32b6601ab82175deafde3c8"} Jul 15 20:21:09.652904 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:09.652875 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-772fb7-mlhr2" event={"ID":"d28d4f7e-710c-4409-88c1-ad1610a7d930","Type":"ContainerStarted","Data":"6baaa8a628ddb03294789694013511c78239f13b798d3d5eb55faaedbdd7a22c"} Jul 15 20:21:09.668348 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:09.668296 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-772fb7-mlhr2" podStartSLOduration=1.668280829 podStartE2EDuration="1.668280829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:21:09.667731478 +0000 UTC m=+2583.912237940" watchObservedRunningTime="2026-07-15 20:21:09.668280829 +0000 UTC m=+2583.912787290" Jul 15 20:21:10.660346 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:10.660309 2554 generic.go:356] "Generic (PLEG): container finished" podID="d28d4f7e-710c-4409-88c1-ad1610a7d930" containerID="7318210c29f289f0c19e1887a4c339d96fed56cba32b6601ab82175deafde3c8" exitCode=0 Jul 15 20:21:10.660727 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:10.660395 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-772fb7-mlhr2" event={"ID":"d28d4f7e-710c-4409-88c1-ad1610a7d930","Type":"ContainerDied","Data":"7318210c29f289f0c19e1887a4c339d96fed56cba32b6601ab82175deafde3c8"} Jul 15 20:21:11.802301 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:11.802271 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-772fb7-mlhr2" Jul 15 20:21:11.938734 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:11.938696 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d28d4f7e-710c-4409-88c1-ad1610a7d930-maas-api-service-ca\") pod \"d28d4f7e-710c-4409-88c1-ad1610a7d930\" (UID: \"d28d4f7e-710c-4409-88c1-ad1610a7d930\") " Jul 15 20:21:11.939093 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:11.939069 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28d4f7e-710c-4409-88c1-ad1610a7d930-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d28d4f7e-710c-4409-88c1-ad1610a7d930" (UID: "d28d4f7e-710c-4409-88c1-ad1610a7d930"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:21:12.039729 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:12.039690 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d28d4f7e-710c-4409-88c1-ad1610a7d930-maas-api-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:21:12.672956 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:12.672911 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-772fb7-mlhr2" event={"ID":"d28d4f7e-710c-4409-88c1-ad1610a7d930","Type":"ContainerDied","Data":"6baaa8a628ddb03294789694013511c78239f13b798d3d5eb55faaedbdd7a22c"} Jul 15 20:21:12.672956 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:12.672936 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-772fb7-mlhr2" Jul 15 20:21:12.673172 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:12.672973 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6baaa8a628ddb03294789694013511c78239f13b798d3d5eb55faaedbdd7a22c" Jul 15 20:21:13.457290 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.457265 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:21:13.554703 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.554601 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-istio-data\") pod \"db02b3f4-8996-4918-acfe-818617c4866a\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " Jul 15 20:21:13.554703 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.554648 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjvpt\" (UniqueName: \"kubernetes.io/projected/db02b3f4-8996-4918-acfe-818617c4866a-kube-api-access-wjvpt\") pod \"db02b3f4-8996-4918-acfe-818617c4866a\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " Jul 15 20:21:13.554703 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.554677 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-credential-socket\") pod \"db02b3f4-8996-4918-acfe-818617c4866a\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " Jul 15 20:21:13.555059 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.554721 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-workload-certs\") pod \"db02b3f4-8996-4918-acfe-818617c4866a\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " Jul 15 20:21:13.555059 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.554778 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-istio-envoy\") pod \"db02b3f4-8996-4918-acfe-818617c4866a\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " Jul 15 20:21:13.555059 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.554808 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-workload-socket\") pod \"db02b3f4-8996-4918-acfe-818617c4866a\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " Jul 15 20:21:13.555059 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.554845 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db02b3f4-8996-4918-acfe-818617c4866a-istio-token\") pod \"db02b3f4-8996-4918-acfe-818617c4866a\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " Jul 15 20:21:13.555059 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.554889 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db02b3f4-8996-4918-acfe-818617c4866a-istiod-ca-cert\") pod \"db02b3f4-8996-4918-acfe-818617c4866a\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " Jul 15 20:21:13.555059 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.554917 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db02b3f4-8996-4918-acfe-818617c4866a-istio-podinfo\") pod \"db02b3f4-8996-4918-acfe-818617c4866a\" (UID: \"db02b3f4-8996-4918-acfe-818617c4866a\") " Jul 15 20:21:13.555345 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.555070 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "db02b3f4-8996-4918-acfe-818617c4866a" (UID: "db02b3f4-8996-4918-acfe-818617c4866a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:21:13.555435 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.555141 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "db02b3f4-8996-4918-acfe-818617c4866a" (UID: "db02b3f4-8996-4918-acfe-818617c4866a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:21:13.555435 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.555208 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "db02b3f4-8996-4918-acfe-818617c4866a" (UID: "db02b3f4-8996-4918-acfe-818617c4866a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:21:13.555545 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.555439 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db02b3f4-8996-4918-acfe-818617c4866a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "db02b3f4-8996-4918-acfe-818617c4866a" (UID: "db02b3f4-8996-4918-acfe-818617c4866a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:21:13.555618 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.555592 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-istio-data" (OuterVolumeSpecName: "istio-data") pod "db02b3f4-8996-4918-acfe-818617c4866a" (UID: "db02b3f4-8996-4918-acfe-818617c4866a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:21:13.557294 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.557259 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db02b3f4-8996-4918-acfe-818617c4866a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "db02b3f4-8996-4918-acfe-818617c4866a" (UID: "db02b3f4-8996-4918-acfe-818617c4866a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:21:13.557403 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.557380 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db02b3f4-8996-4918-acfe-818617c4866a-istio-token" (OuterVolumeSpecName: "istio-token") pod "db02b3f4-8996-4918-acfe-818617c4866a" (UID: "db02b3f4-8996-4918-acfe-818617c4866a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:21:13.557568 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.557549 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "db02b3f4-8996-4918-acfe-818617c4866a" (UID: "db02b3f4-8996-4918-acfe-818617c4866a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:21:13.557645 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.557625 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db02b3f4-8996-4918-acfe-818617c4866a-kube-api-access-wjvpt" (OuterVolumeSpecName: "kube-api-access-wjvpt") pod "db02b3f4-8996-4918-acfe-818617c4866a" (UID: "db02b3f4-8996-4918-acfe-818617c4866a"). InnerVolumeSpecName "kube-api-access-wjvpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:21:13.656187 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.656149 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-istio-data\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:21:13.656187 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.656185 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjvpt\" (UniqueName: \"kubernetes.io/projected/db02b3f4-8996-4918-acfe-818617c4866a-kube-api-access-wjvpt\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:21:13.656415 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.656200 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-credential-socket\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:21:13.656415 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.656213 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-workload-certs\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:21:13.656415 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.656225 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-istio-envoy\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:21:13.656415 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.656235 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db02b3f4-8996-4918-acfe-818617c4866a-workload-socket\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:21:13.656415 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.656245 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db02b3f4-8996-4918-acfe-818617c4866a-istio-token\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:21:13.656415 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.656255 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db02b3f4-8996-4918-acfe-818617c4866a-istiod-ca-cert\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:21:13.656415 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.656266 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db02b3f4-8996-4918-acfe-818617c4866a-istio-podinfo\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:21:13.680271 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.680238 2554 generic.go:356] "Generic (PLEG): container finished" podID="db02b3f4-8996-4918-acfe-818617c4866a" containerID="03ef748925382288a8a9ba4a09d19b4e05d9aa7612b77f967b56ed743ac67970" exitCode=0 Jul 15 20:21:13.680456 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.680291 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" Jul 15 20:21:13.680456 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.680322 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" event={"ID":"db02b3f4-8996-4918-acfe-818617c4866a","Type":"ContainerDied","Data":"03ef748925382288a8a9ba4a09d19b4e05d9aa7612b77f967b56ed743ac67970"} Jul 15 20:21:13.680456 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.680386 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9" event={"ID":"db02b3f4-8996-4918-acfe-818617c4866a","Type":"ContainerDied","Data":"d97562cad3185110a666fbb951a337eee2d4d33063666930406ccd472d2a4765"} Jul 15 20:21:13.680456 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.680416 2554 scope.go:117] "RemoveContainer" containerID="03ef748925382288a8a9ba4a09d19b4e05d9aa7612b77f967b56ed743ac67970" Jul 15 20:21:13.690118 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.690100 2554 scope.go:117] "RemoveContainer" containerID="607a7368bc32f25efeaaea71d2cf530305d2a78e81804a496337dafeb8e368b6" Jul 15 20:21:13.699438 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.699420 2554 scope.go:117] "RemoveContainer" containerID="03ef748925382288a8a9ba4a09d19b4e05d9aa7612b77f967b56ed743ac67970" Jul 15 20:21:13.699721 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:21:13.699701 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03ef748925382288a8a9ba4a09d19b4e05d9aa7612b77f967b56ed743ac67970\": container with ID starting with 03ef748925382288a8a9ba4a09d19b4e05d9aa7612b77f967b56ed743ac67970 not found: ID does not exist" containerID="03ef748925382288a8a9ba4a09d19b4e05d9aa7612b77f967b56ed743ac67970" Jul 15 20:21:13.699810 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.699727 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03ef748925382288a8a9ba4a09d19b4e05d9aa7612b77f967b56ed743ac67970"} err="failed to get container status \"03ef748925382288a8a9ba4a09d19b4e05d9aa7612b77f967b56ed743ac67970\": rpc error: code = NotFound desc = could not find container \"03ef748925382288a8a9ba4a09d19b4e05d9aa7612b77f967b56ed743ac67970\": container with ID starting with 03ef748925382288a8a9ba4a09d19b4e05d9aa7612b77f967b56ed743ac67970 not found: ID does not exist" Jul 15 20:21:13.699810 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.699744 2554 scope.go:117] "RemoveContainer" containerID="607a7368bc32f25efeaaea71d2cf530305d2a78e81804a496337dafeb8e368b6" Jul 15 20:21:13.700004 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:21:13.699987 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"607a7368bc32f25efeaaea71d2cf530305d2a78e81804a496337dafeb8e368b6\": container with ID starting with 607a7368bc32f25efeaaea71d2cf530305d2a78e81804a496337dafeb8e368b6 not found: ID does not exist" containerID="607a7368bc32f25efeaaea71d2cf530305d2a78e81804a496337dafeb8e368b6" Jul 15 20:21:13.700045 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.700010 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"607a7368bc32f25efeaaea71d2cf530305d2a78e81804a496337dafeb8e368b6"} err="failed to get container status \"607a7368bc32f25efeaaea71d2cf530305d2a78e81804a496337dafeb8e368b6\": rpc error: code = NotFound desc = could not find container \"607a7368bc32f25efeaaea71d2cf530305d2a78e81804a496337dafeb8e368b6\": container with ID starting with 607a7368bc32f25efeaaea71d2cf530305d2a78e81804a496337dafeb8e368b6 not found: ID does not exist" Jul 15 20:21:13.715097 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.715073 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9"] Jul 15 20:21:13.725328 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:13.725306 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cae23a-openshift-default-96c54fcc7-f9sh9"] Jul 15 20:21:14.239677 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:14.239642 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db02b3f4-8996-4918-acfe-818617c4866a" path="/var/lib/kubelet/pods/db02b3f4-8996-4918-acfe-818617c4866a/volumes" Jul 15 20:21:19.150545 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.150513 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t"] Jul 15 20:21:19.151049 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.150709 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" podUID="2d056ca7-4fe4-4787-9716-242dae9ac81b" containerName="maas-api" containerID="cri-o://e2ab0e864a79dbf9f59cd3b4b97ff83eb07f11ece3e247dc74bbaac0c837bcac" gracePeriod=30 Jul 15 20:21:19.173491 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.173466 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:19.395092 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.395065 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" Jul 15 20:21:19.511267 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.511237 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d056ca7-4fe4-4787-9716-242dae9ac81b-openshift-service-ca\") pod \"2d056ca7-4fe4-4787-9716-242dae9ac81b\" (UID: \"2d056ca7-4fe4-4787-9716-242dae9ac81b\") " Jul 15 20:21:19.511454 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.511292 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pmhl\" (UniqueName: \"kubernetes.io/projected/2d056ca7-4fe4-4787-9716-242dae9ac81b-kube-api-access-9pmhl\") pod \"2d056ca7-4fe4-4787-9716-242dae9ac81b\" (UID: \"2d056ca7-4fe4-4787-9716-242dae9ac81b\") " Jul 15 20:21:19.511454 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.511312 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d056ca7-4fe4-4787-9716-242dae9ac81b-maas-api-tls\") pod \"2d056ca7-4fe4-4787-9716-242dae9ac81b\" (UID: \"2d056ca7-4fe4-4787-9716-242dae9ac81b\") " Jul 15 20:21:19.511682 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.511648 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d056ca7-4fe4-4787-9716-242dae9ac81b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2d056ca7-4fe4-4787-9716-242dae9ac81b" (UID: "2d056ca7-4fe4-4787-9716-242dae9ac81b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:21:19.513550 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.513527 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d056ca7-4fe4-4787-9716-242dae9ac81b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2d056ca7-4fe4-4787-9716-242dae9ac81b" (UID: "2d056ca7-4fe4-4787-9716-242dae9ac81b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:21:19.513659 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.513570 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d056ca7-4fe4-4787-9716-242dae9ac81b-kube-api-access-9pmhl" (OuterVolumeSpecName: "kube-api-access-9pmhl") pod "2d056ca7-4fe4-4787-9716-242dae9ac81b" (UID: "2d056ca7-4fe4-4787-9716-242dae9ac81b"). InnerVolumeSpecName "kube-api-access-9pmhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:21:19.612584 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.612544 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d056ca7-4fe4-4787-9716-242dae9ac81b-openshift-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:21:19.612584 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.612576 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pmhl\" (UniqueName: \"kubernetes.io/projected/2d056ca7-4fe4-4787-9716-242dae9ac81b-kube-api-access-9pmhl\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:21:19.612584 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.612586 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d056ca7-4fe4-4787-9716-242dae9ac81b-maas-api-tls\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:21:19.715336 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.715298 2554 generic.go:356] "Generic (PLEG): container finished" podID="2d056ca7-4fe4-4787-9716-242dae9ac81b" containerID="e2ab0e864a79dbf9f59cd3b4b97ff83eb07f11ece3e247dc74bbaac0c837bcac" exitCode=0 Jul 15 20:21:19.715336 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.715335 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" Jul 15 20:21:19.715550 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.715383 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" event={"ID":"2d056ca7-4fe4-4787-9716-242dae9ac81b","Type":"ContainerDied","Data":"e2ab0e864a79dbf9f59cd3b4b97ff83eb07f11ece3e247dc74bbaac0c837bcac"} Jul 15 20:21:19.715550 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.715438 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t" event={"ID":"2d056ca7-4fe4-4787-9716-242dae9ac81b","Type":"ContainerDied","Data":"45afc4186d34774da01023f3e2c614793417468233c12056412dbaf670898e97"} Jul 15 20:21:19.715550 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.715462 2554 scope.go:117] "RemoveContainer" containerID="e2ab0e864a79dbf9f59cd3b4b97ff83eb07f11ece3e247dc74bbaac0c837bcac" Jul 15 20:21:19.725161 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.725136 2554 scope.go:117] "RemoveContainer" containerID="e2ab0e864a79dbf9f59cd3b4b97ff83eb07f11ece3e247dc74bbaac0c837bcac" Jul 15 20:21:19.725467 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:21:19.725448 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ab0e864a79dbf9f59cd3b4b97ff83eb07f11ece3e247dc74bbaac0c837bcac\": container with ID starting with e2ab0e864a79dbf9f59cd3b4b97ff83eb07f11ece3e247dc74bbaac0c837bcac not found: ID does not exist" containerID="e2ab0e864a79dbf9f59cd3b4b97ff83eb07f11ece3e247dc74bbaac0c837bcac" Jul 15 20:21:19.725518 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.725478 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ab0e864a79dbf9f59cd3b4b97ff83eb07f11ece3e247dc74bbaac0c837bcac"} err="failed to get container status \"e2ab0e864a79dbf9f59cd3b4b97ff83eb07f11ece3e247dc74bbaac0c837bcac\": rpc error: code = NotFound desc = could not find container \"e2ab0e864a79dbf9f59cd3b4b97ff83eb07f11ece3e247dc74bbaac0c837bcac\": container with ID starting with e2ab0e864a79dbf9f59cd3b4b97ff83eb07f11ece3e247dc74bbaac0c837bcac not found: ID does not exist" Jul 15 20:21:19.737644 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.737614 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t"] Jul 15 20:21:19.741640 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:19.741614 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-772fb7-6db4947f95-4mk5t"] Jul 15 20:21:20.238806 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:20.238775 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d056ca7-4fe4-4787-9716-242dae9ac81b" path="/var/lib/kubelet/pods/2d056ca7-4fe4-4787-9716-242dae9ac81b/volumes" Jul 15 20:21:22.227213 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:22.227180 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:21:23.680589 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:23.680557 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:29.553501 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:29.553418 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:29.624480 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:29.624400 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:34.824730 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:34.824696 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:34.932349 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:34.932313 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:35.026279 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.026247 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:35.364670 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.364630 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft"] Jul 15 20:21:35.365144 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.365129 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db02b3f4-8996-4918-acfe-818617c4866a" containerName="istio-proxy" Jul 15 20:21:35.365193 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.365148 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="db02b3f4-8996-4918-acfe-818617c4866a" containerName="istio-proxy" Jul 15 20:21:35.365193 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.365160 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d056ca7-4fe4-4787-9716-242dae9ac81b" containerName="maas-api" Jul 15 20:21:35.365193 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.365169 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d056ca7-4fe4-4787-9716-242dae9ac81b" containerName="maas-api" Jul 15 20:21:35.365193 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.365186 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d28d4f7e-710c-4409-88c1-ad1610a7d930" containerName="revoke-keys" Jul 15 20:21:35.365319 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.365194 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d28d4f7e-710c-4409-88c1-ad1610a7d930" containerName="revoke-keys" Jul 15 20:21:35.365319 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.365220 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db02b3f4-8996-4918-acfe-818617c4866a" containerName="istio-proxy" Jul 15 20:21:35.365319 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.365228 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="db02b3f4-8996-4918-acfe-818617c4866a" containerName="istio-proxy" Jul 15 20:21:35.365319 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.365302 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="db02b3f4-8996-4918-acfe-818617c4866a" containerName="istio-proxy" Jul 15 20:21:35.365319 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.365313 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d28d4f7e-710c-4409-88c1-ad1610a7d930" containerName="revoke-keys" Jul 15 20:21:35.365463 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.365323 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="db02b3f4-8996-4918-acfe-818617c4866a" containerName="istio-proxy" Jul 15 20:21:35.365463 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.365335 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d056ca7-4fe4-4787-9716-242dae9ac81b" containerName="maas-api" Jul 15 20:21:35.370288 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.370264 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.373536 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.373514 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-3ada3b-openshift-default-dockercfg-d75fb\"" Jul 15 20:21:35.402422 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.402388 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft"] Jul 15 20:21:35.485746 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.485709 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:35.545689 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.545656 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-istio-envoy\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.545689 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.545698 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs2w7\" (UniqueName: \"kubernetes.io/projected/8b73a338-a660-472b-8b02-ece25a4b953a-kube-api-access-hs2w7\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.545939 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.545723 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b73a338-a660-472b-8b02-ece25a4b953a-istio-podinfo\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.545939 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.545781 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-credential-socket\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.545939 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.545816 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-istio-data\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.545939 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.545838 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-workload-socket\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.545939 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.545859 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b73a338-a660-472b-8b02-ece25a4b953a-istio-token\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.545939 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.545879 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b73a338-a660-472b-8b02-ece25a4b953a-istiod-ca-cert\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.545939 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.545901 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-workload-certs\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.638316 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.638229 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:35.646254 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.646225 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-workload-certs\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.646426 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.646294 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-istio-envoy\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.646426 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.646331 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs2w7\" (UniqueName: \"kubernetes.io/projected/8b73a338-a660-472b-8b02-ece25a4b953a-kube-api-access-hs2w7\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.646426 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.646367 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b73a338-a660-472b-8b02-ece25a4b953a-istio-podinfo\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.646426 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.646397 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-credential-socket\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.646644 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.646435 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-istio-data\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.646644 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.646472 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-workload-socket\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.646644 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.646500 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b73a338-a660-472b-8b02-ece25a4b953a-istio-token\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.646644 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.646527 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b73a338-a660-472b-8b02-ece25a4b953a-istiod-ca-cert\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.646882 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.646679 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-workload-certs\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.646882 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.646794 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-credential-socket\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.646882 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.646810 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-istio-data\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.647040 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.646937 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-workload-socket\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.647213 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.647195 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b73a338-a660-472b-8b02-ece25a4b953a-istiod-ca-cert\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.649001 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.648975 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-istio-envoy\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.649198 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.649183 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b73a338-a660-472b-8b02-ece25a4b953a-istio-podinfo\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.659274 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.659186 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b73a338-a660-472b-8b02-ece25a4b953a-istio-token\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.660947 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.660926 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs2w7\" (UniqueName: \"kubernetes.io/projected/8b73a338-a660-472b-8b02-ece25a4b953a-kube-api-access-hs2w7\") pod \"e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.680842 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.680810 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:35.837412 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.837380 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft"] Jul 15 20:21:35.838599 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:21:35.838567 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b73a338_a660_472b_8b02_ece25a4b953a.slice/crio-ad3a9a16f94435973fbf149d06aac05259687e7d00baa717c41ce1f76451ee50 WatchSource:0}: Error finding container ad3a9a16f94435973fbf149d06aac05259687e7d00baa717c41ce1f76451ee50: Status 404 returned error can't find the container with id ad3a9a16f94435973fbf149d06aac05259687e7d00baa717c41ce1f76451ee50 Jul 15 20:21:35.841250 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.841206 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:21:35.841504 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.841474 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:21:35.841692 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:35.841669 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:21:36.812457 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:36.812417 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" event={"ID":"8b73a338-a660-472b-8b02-ece25a4b953a","Type":"ContainerStarted","Data":"62ed9bce1a8740ebc2404925780740873b58a02f30a122833431b3a23706122b"} Jul 15 20:21:36.812634 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:36.812472 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" event={"ID":"8b73a338-a660-472b-8b02-ece25a4b953a","Type":"ContainerStarted","Data":"ad3a9a16f94435973fbf149d06aac05259687e7d00baa717c41ce1f76451ee50"} Jul 15 20:21:36.850056 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:36.849999 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" podStartSLOduration=1.8499822240000001 podStartE2EDuration="1.849982224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:21:36.848572502 +0000 UTC m=+2611.093078962" watchObservedRunningTime="2026-07-15 20:21:36.849982224 +0000 UTC m=+2611.094488684" Jul 15 20:21:37.682016 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:37.681979 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:37.687584 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:37.687556 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:37.818579 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:37.818542 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:37.819429 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:37.819406 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:38.847207 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:38.847176 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:38.931678 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:38.931645 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc"] Jul 15 20:21:38.935246 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:38.935222 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" Jul 15 20:21:38.938199 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:38.938173 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-3ada3b-serving-cert\"" Jul 15 20:21:38.938633 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:38.938613 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c5cgj\"" Jul 15 20:21:38.938791 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:38.938660 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 20:21:38.938868 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:38.938797 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 20:21:38.951246 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:38.951214 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:38.952313 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:38.952288 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc"] Jul 15 20:21:38.972164 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:38.972130 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1168f03a-3a28-4e50-8020-c53b1fb33bb9-maas-api-tls\") pod \"maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc\" (UID: \"1168f03a-3a28-4e50-8020-c53b1fb33bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" Jul 15 20:21:38.972330 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:38.972181 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68vks\" (UniqueName: \"kubernetes.io/projected/1168f03a-3a28-4e50-8020-c53b1fb33bb9-kube-api-access-68vks\") pod \"maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc\" (UID: \"1168f03a-3a28-4e50-8020-c53b1fb33bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" Jul 15 20:21:38.972330 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:38.972239 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1168f03a-3a28-4e50-8020-c53b1fb33bb9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc\" (UID: \"1168f03a-3a28-4e50-8020-c53b1fb33bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" Jul 15 20:21:39.052860 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:39.052827 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:39.073060 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:39.073024 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1168f03a-3a28-4e50-8020-c53b1fb33bb9-maas-api-tls\") pod \"maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc\" (UID: \"1168f03a-3a28-4e50-8020-c53b1fb33bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" Jul 15 20:21:39.073207 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:39.073069 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vks\" (UniqueName: \"kubernetes.io/projected/1168f03a-3a28-4e50-8020-c53b1fb33bb9-kube-api-access-68vks\") pod \"maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc\" (UID: \"1168f03a-3a28-4e50-8020-c53b1fb33bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" Jul 15 20:21:39.073207 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:39.073115 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1168f03a-3a28-4e50-8020-c53b1fb33bb9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc\" (UID: \"1168f03a-3a28-4e50-8020-c53b1fb33bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" Jul 15 20:21:39.073207 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:21:39.073186 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-serving-cert: secret "maas-api-e2e-shared-a-3ada3b-serving-cert" not found Jul 15 20:21:39.073318 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:21:39.073277 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1168f03a-3a28-4e50-8020-c53b1fb33bb9-maas-api-tls podName:1168f03a-3a28-4e50-8020-c53b1fb33bb9 nodeName:}" failed. No retries permitted until 2026-07-15 20:21:39.573255148 +0000 UTC m=+2613.817761587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1168f03a-3a28-4e50-8020-c53b1fb33bb9-maas-api-tls") pod "maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" (UID: "1168f03a-3a28-4e50-8020-c53b1fb33bb9") : secret "maas-api-e2e-shared-a-3ada3b-serving-cert" not found Jul 15 20:21:39.073817 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:39.073781 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1168f03a-3a28-4e50-8020-c53b1fb33bb9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc\" (UID: \"1168f03a-3a28-4e50-8020-c53b1fb33bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" Jul 15 20:21:39.086672 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:39.086639 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68vks\" (UniqueName: \"kubernetes.io/projected/1168f03a-3a28-4e50-8020-c53b1fb33bb9-kube-api-access-68vks\") pod \"maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc\" (UID: \"1168f03a-3a28-4e50-8020-c53b1fb33bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" Jul 15 20:21:39.318785 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:39.318726 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:39.537870 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:39.537841 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:39.576696 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:39.576597 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1168f03a-3a28-4e50-8020-c53b1fb33bb9-maas-api-tls\") pod \"maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc\" (UID: \"1168f03a-3a28-4e50-8020-c53b1fb33bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" Jul 15 20:21:39.579348 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:39.579316 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1168f03a-3a28-4e50-8020-c53b1fb33bb9-maas-api-tls\") pod \"maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc\" (UID: \"1168f03a-3a28-4e50-8020-c53b1fb33bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" Jul 15 20:21:39.635569 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:39.635531 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:39.847813 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:39.847700 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" Jul 15 20:21:39.997550 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:39.997527 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc"] Jul 15 20:21:39.999268 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:21:39.999240 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1168f03a_3a28_4e50_8020_c53b1fb33bb9.slice/crio-df6b723d488726b7c7b40b41cef40438fe6bbdc84e690ba772490bcf032d579f WatchSource:0}: Error finding container df6b723d488726b7c7b40b41cef40438fe6bbdc84e690ba772490bcf032d579f: Status 404 returned error can't find the container with id df6b723d488726b7c7b40b41cef40438fe6bbdc84e690ba772490bcf032d579f Jul 15 20:21:40.837148 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:40.837048 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" event={"ID":"1168f03a-3a28-4e50-8020-c53b1fb33bb9","Type":"ContainerStarted","Data":"d7365f93c3d5ccde7dae7b7a67b7bfcfb3707d2c806040fc7b220ddd8209921b"} Jul 15 20:21:40.837292 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:40.837211 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" Jul 15 20:21:40.837292 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:40.837249 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" event={"ID":"1168f03a-3a28-4e50-8020-c53b1fb33bb9","Type":"ContainerStarted","Data":"df6b723d488726b7c7b40b41cef40438fe6bbdc84e690ba772490bcf032d579f"} Jul 15 20:21:40.864265 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:40.864200 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" podStartSLOduration=2.367421328 podStartE2EDuration="2.864178138s" totalImagesPullingTime="496.75681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:21:40.858095394 +0000 UTC m=+2615.102601855" watchObservedRunningTime="2026-07-15 20:21:40.864178138 +0000 UTC m=+2615.108684601" Jul 15 20:21:42.833807 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:42.833762 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:21:43.282793 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:43.282739 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:43.624889 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:43.624788 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:44.067918 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:44.067884 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:44.163543 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:44.163509 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:46.850177 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:46.850136 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" Jul 15 20:21:47.305978 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.305943 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:47.371993 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.371960 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv"] Jul 15 20:21:47.375909 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.375884 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" Jul 15 20:21:47.378903 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.378858 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-54098e-serving-cert\"" Jul 15 20:21:47.387291 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.387264 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv"] Jul 15 20:21:47.441279 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.441239 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:47.542801 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.542741 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a67a619-30ec-4583-96d9-7b80e3febe7f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-54098e-6997fc4668-mshbv\" (UID: \"2a67a619-30ec-4583-96d9-7b80e3febe7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" Jul 15 20:21:47.543106 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.543073 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a67a619-30ec-4583-96d9-7b80e3febe7f-maas-api-tls\") pod \"maas-api-e2e-shared-b-54098e-6997fc4668-mshbv\" (UID: \"2a67a619-30ec-4583-96d9-7b80e3febe7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" Jul 15 20:21:47.543329 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.543305 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xfvk\" (UniqueName: \"kubernetes.io/projected/2a67a619-30ec-4583-96d9-7b80e3febe7f-kube-api-access-5xfvk\") pod \"maas-api-e2e-shared-b-54098e-6997fc4668-mshbv\" (UID: \"2a67a619-30ec-4583-96d9-7b80e3febe7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" Jul 15 20:21:47.545014 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.544976 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:47.644980 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.644879 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xfvk\" (UniqueName: \"kubernetes.io/projected/2a67a619-30ec-4583-96d9-7b80e3febe7f-kube-api-access-5xfvk\") pod \"maas-api-e2e-shared-b-54098e-6997fc4668-mshbv\" (UID: \"2a67a619-30ec-4583-96d9-7b80e3febe7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" Jul 15 20:21:47.645169 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.644979 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a67a619-30ec-4583-96d9-7b80e3febe7f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-54098e-6997fc4668-mshbv\" (UID: \"2a67a619-30ec-4583-96d9-7b80e3febe7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" Jul 15 20:21:47.645169 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.645014 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a67a619-30ec-4583-96d9-7b80e3febe7f-maas-api-tls\") pod \"maas-api-e2e-shared-b-54098e-6997fc4668-mshbv\" (UID: \"2a67a619-30ec-4583-96d9-7b80e3febe7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" Jul 15 20:21:47.645878 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.645843 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a67a619-30ec-4583-96d9-7b80e3febe7f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-54098e-6997fc4668-mshbv\" (UID: \"2a67a619-30ec-4583-96d9-7b80e3febe7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" Jul 15 20:21:47.647576 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.647550 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a67a619-30ec-4583-96d9-7b80e3febe7f-maas-api-tls\") pod \"maas-api-e2e-shared-b-54098e-6997fc4668-mshbv\" (UID: \"2a67a619-30ec-4583-96d9-7b80e3febe7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" Jul 15 20:21:47.656914 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.656889 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xfvk\" (UniqueName: \"kubernetes.io/projected/2a67a619-30ec-4583-96d9-7b80e3febe7f-kube-api-access-5xfvk\") pod \"maas-api-e2e-shared-b-54098e-6997fc4668-mshbv\" (UID: \"2a67a619-30ec-4583-96d9-7b80e3febe7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" Jul 15 20:21:47.688892 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.688858 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" Jul 15 20:21:47.691379 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.691354 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:47.874241 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.872120 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv"] Jul 15 20:21:47.876702 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:21:47.876672 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a67a619_30ec_4583_96d9_7b80e3febe7f.slice/crio-74e681d973b362d77d9b73a5481c29e183585f6f8ce8ad96865ba0a623636226 WatchSource:0}: Error finding container 74e681d973b362d77d9b73a5481c29e183585f6f8ce8ad96865ba0a623636226: Status 404 returned error can't find the container with id 74e681d973b362d77d9b73a5481c29e183585f6f8ce8ad96865ba0a623636226 Jul 15 20:21:47.990465 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:47.990428 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:48.091593 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:48.091563 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:48.888191 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:48.888155 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" event={"ID":"2a67a619-30ec-4583-96d9-7b80e3febe7f","Type":"ContainerStarted","Data":"be8c80b9e71d3f599a816acf2fdcc6e93766a4d6b0a953f05f348280fceaf1b3"} Jul 15 20:21:48.888642 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:48.888215 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" event={"ID":"2a67a619-30ec-4583-96d9-7b80e3febe7f","Type":"ContainerStarted","Data":"74e681d973b362d77d9b73a5481c29e183585f6f8ce8ad96865ba0a623636226"} Jul 15 20:21:48.888642 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:48.888247 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" Jul 15 20:21:48.910330 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:48.910268 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" podStartSLOduration=0.995278544 podStartE2EDuration="1.910252082s" totalImagesPullingTime="914.973538ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:21:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:21:48.907865728 +0000 UTC m=+2623.152372190" watchObservedRunningTime="2026-07-15 20:21:48.910252082 +0000 UTC m=+2623.154758583" Jul 15 20:21:50.901716 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:50.901637 2554 generic.go:356] "Generic (PLEG): container finished" podID="8b73a338-a660-472b-8b02-ece25a4b953a" containerID="62ed9bce1a8740ebc2404925780740873b58a02f30a122833431b3a23706122b" exitCode=0 Jul 15 20:21:50.902221 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:50.901707 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" event={"ID":"8b73a338-a660-472b-8b02-ece25a4b953a","Type":"ContainerDied","Data":"62ed9bce1a8740ebc2404925780740873b58a02f30a122833431b3a23706122b"} Jul 15 20:21:50.902304 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:50.902239 2554 scope.go:117] "RemoveContainer" containerID="62ed9bce1a8740ebc2404925780740873b58a02f30a122833431b3a23706122b" Jul 15 20:21:50.903285 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:50.903260 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:21:50.903351 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:50.903336 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:21:50.903395 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:50.903368 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 15 20:21:51.726636 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:51.726606 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:21:51.909550 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:51.909513 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" event={"ID":"8b73a338-a660-472b-8b02-ece25a4b953a","Type":"ContainerStarted","Data":"bf92c3273ada452809da409cfa532202ab1dbe2e1db4671e81d0402281fd9a12"} Jul 15 20:21:52.681164 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:52.681124 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:52.682301 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:52.682269 2554 patch_prober.go:28] interesting pod/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 15 20:21:52.682452 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:52.682322 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" podUID="8b73a338-a660-472b-8b02-ece25a4b953a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 15 20:21:52.822779 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:52.818845 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:53.681616 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:53.681576 2554 patch_prober.go:28] interesting pod/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 15 20:21:53.682059 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:53.681634 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" podUID="8b73a338-a660-472b-8b02-ece25a4b953a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 15 20:21:54.681384 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:54.681347 2554 patch_prober.go:28] interesting pod/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 15 20:21:54.681569 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:54.681405 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" podUID="8b73a338-a660-472b-8b02-ece25a4b953a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 15 20:21:54.899859 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:54.899826 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" Jul 15 20:21:55.635953 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:55.635903 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:21:55.681250 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:55.681213 2554 patch_prober.go:28] interesting pod/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 15 20:21:55.681417 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:55.681270 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" podUID="8b73a338-a660-472b-8b02-ece25a4b953a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 15 20:21:56.685655 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:56.685623 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:21:56.686780 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:21:56.686741 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:22:00.326735 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:22:00.326690 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:22:00.426973 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:22:00.426933 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:22:01.151405 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:22:01.151371 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:22:06.140609 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:22:06.140517 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:22:08.861354 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:22:08.861312 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:22:08.962114 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:22:08.962077 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:22:32.097832 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:22:32.097793 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:22:35.942708 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:22:35.942673 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:22:49.333971 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:22:49.333931 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:23:06.131928 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:23:06.131835 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:23:06.463907 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:23:06.463879 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 20:23:06.468405 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:23:06.468383 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 20:23:06.479781 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:23:06.479739 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 20:23:06.484225 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:23:06.484204 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 20:23:21.831525 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:23:21.831492 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:23:40.161697 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:23:40.161650 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:23:40.263454 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:23:40.263417 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:23:40.738121 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:23:40.738087 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:23:56.040915 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:23:56.040867 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:24:11.134679 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:24:11.134643 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:24:22.727852 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:24:22.727811 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:24:23.699173 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:24:23.699132 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:24:26.845821 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:24:26.845775 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:24:39.545306 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:24:39.545269 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:24:46.382485 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:24:46.382447 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:24:48.654398 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:24:48.654359 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:24:51.290982 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:24:51.290939 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:25:02.350060 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:25:02.350030 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:25:06.747777 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:25:06.747723 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:25:26.037333 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:25:26.037299 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:25:36.122451 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:25:36.122409 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:25:39.734650 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:25:39.729608 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:25:45.981883 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:25:45.981851 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:25:46.727495 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:25:46.727458 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:25:53.984535 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:25:53.984502 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:25:58.229686 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:25:58.229605 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:26:20.606242 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:26:20.606215 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:26:24.834290 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:26:24.834250 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:26:33.878152 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:26:33.878117 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:26:33.946102 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:26:33.946068 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:26:43.135168 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:26:43.135119 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:27:01.118447 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:01.118411 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:01.218976 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:01.218944 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:02.335563 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:02.335527 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:27:06.009787 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:06.009717 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:06.882714 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:06.882620 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:09.664785 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:09.664740 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:09.765028 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:09.764991 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:18.827019 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:18.826986 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:27:27.041008 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:27.040906 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:27.141068 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:27.141024 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:36.795796 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:36.795742 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc"] Jul 15 20:27:36.796339 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:36.795981 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" podUID="1168f03a-3a28-4e50-8020-c53b1fb33bb9" containerName="maas-api" containerID="cri-o://d7365f93c3d5ccde7dae7b7a67b7bfcfb3707d2c806040fc7b220ddd8209921b" gracePeriod=30 Jul 15 20:27:36.826955 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:36.826931 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:36.844237 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:36.844201 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" podUID="1168f03a-3a28-4e50-8020-c53b1fb33bb9" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.70:8443/health\": dial tcp 10.132.0.70:8443: connect: connection refused" Jul 15 20:27:37.047086 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:37.047025 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" Jul 15 20:27:37.213643 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:37.213600 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68vks\" (UniqueName: \"kubernetes.io/projected/1168f03a-3a28-4e50-8020-c53b1fb33bb9-kube-api-access-68vks\") pod \"1168f03a-3a28-4e50-8020-c53b1fb33bb9\" (UID: \"1168f03a-3a28-4e50-8020-c53b1fb33bb9\") " Jul 15 20:27:37.213880 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:37.213693 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1168f03a-3a28-4e50-8020-c53b1fb33bb9-openshift-service-ca\") pod \"1168f03a-3a28-4e50-8020-c53b1fb33bb9\" (UID: \"1168f03a-3a28-4e50-8020-c53b1fb33bb9\") " Jul 15 20:27:37.213880 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:37.213719 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1168f03a-3a28-4e50-8020-c53b1fb33bb9-maas-api-tls\") pod \"1168f03a-3a28-4e50-8020-c53b1fb33bb9\" (UID: \"1168f03a-3a28-4e50-8020-c53b1fb33bb9\") " Jul 15 20:27:37.214110 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:37.214079 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1168f03a-3a28-4e50-8020-c53b1fb33bb9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1168f03a-3a28-4e50-8020-c53b1fb33bb9" (UID: "1168f03a-3a28-4e50-8020-c53b1fb33bb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:27:37.215942 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:37.215906 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1168f03a-3a28-4e50-8020-c53b1fb33bb9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1168f03a-3a28-4e50-8020-c53b1fb33bb9" (UID: "1168f03a-3a28-4e50-8020-c53b1fb33bb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:27:37.216058 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:37.215972 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1168f03a-3a28-4e50-8020-c53b1fb33bb9-kube-api-access-68vks" (OuterVolumeSpecName: "kube-api-access-68vks") pod "1168f03a-3a28-4e50-8020-c53b1fb33bb9" (UID: "1168f03a-3a28-4e50-8020-c53b1fb33bb9"). InnerVolumeSpecName "kube-api-access-68vks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:27:37.314813 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:37.314697 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1168f03a-3a28-4e50-8020-c53b1fb33bb9-openshift-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:27:37.314813 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:37.314729 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1168f03a-3a28-4e50-8020-c53b1fb33bb9-maas-api-tls\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:27:37.314813 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:37.314739 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68vks\" (UniqueName: \"kubernetes.io/projected/1168f03a-3a28-4e50-8020-c53b1fb33bb9-kube-api-access-68vks\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:27:37.868645 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:37.868609 2554 generic.go:356] "Generic (PLEG): container finished" podID="1168f03a-3a28-4e50-8020-c53b1fb33bb9" containerID="d7365f93c3d5ccde7dae7b7a67b7bfcfb3707d2c806040fc7b220ddd8209921b" exitCode=0 Jul 15 20:27:37.869098 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:37.868653 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" event={"ID":"1168f03a-3a28-4e50-8020-c53b1fb33bb9","Type":"ContainerDied","Data":"d7365f93c3d5ccde7dae7b7a67b7bfcfb3707d2c806040fc7b220ddd8209921b"} Jul 15 20:27:37.869098 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:37.868657 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" Jul 15 20:27:37.869098 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:37.868700 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc" event={"ID":"1168f03a-3a28-4e50-8020-c53b1fb33bb9","Type":"ContainerDied","Data":"df6b723d488726b7c7b40b41cef40438fe6bbdc84e690ba772490bcf032d579f"} Jul 15 20:27:37.869098 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:37.868727 2554 scope.go:117] "RemoveContainer" containerID="d7365f93c3d5ccde7dae7b7a67b7bfcfb3707d2c806040fc7b220ddd8209921b" Jul 15 20:27:37.879039 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:37.879021 2554 scope.go:117] "RemoveContainer" containerID="d7365f93c3d5ccde7dae7b7a67b7bfcfb3707d2c806040fc7b220ddd8209921b" Jul 15 20:27:37.879342 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:27:37.879323 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7365f93c3d5ccde7dae7b7a67b7bfcfb3707d2c806040fc7b220ddd8209921b\": container with ID starting with d7365f93c3d5ccde7dae7b7a67b7bfcfb3707d2c806040fc7b220ddd8209921b not found: ID does not exist" containerID="d7365f93c3d5ccde7dae7b7a67b7bfcfb3707d2c806040fc7b220ddd8209921b" Jul 15 20:27:37.879412 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:37.879356 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7365f93c3d5ccde7dae7b7a67b7bfcfb3707d2c806040fc7b220ddd8209921b"} err="failed to get container status \"d7365f93c3d5ccde7dae7b7a67b7bfcfb3707d2c806040fc7b220ddd8209921b\": rpc error: code = NotFound desc = could not find container \"d7365f93c3d5ccde7dae7b7a67b7bfcfb3707d2c806040fc7b220ddd8209921b\": container with ID starting with d7365f93c3d5ccde7dae7b7a67b7bfcfb3707d2c806040fc7b220ddd8209921b not found: ID does not exist" Jul 15 20:27:37.916772 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:37.916725 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc"] Jul 15 20:27:37.920725 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:37.920684 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3ada3b-775867895b-s7fjc"] Jul 15 20:27:38.238183 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:38.238153 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1168f03a-3a28-4e50-8020-c53b1fb33bb9" path="/var/lib/kubelet/pods/1168f03a-3a28-4e50-8020-c53b1fb33bb9/volumes" Jul 15 20:27:40.049245 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:40.049208 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:27:41.444515 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:41.444484 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:47.629029 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:47.628965 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:47.706219 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:47.706185 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:52.133987 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:52.133946 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft"] Jul 15 20:27:52.134449 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:52.134226 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" podUID="8b73a338-a660-472b-8b02-ece25a4b953a" containerName="istio-proxy" containerID="cri-o://bf92c3273ada452809da409cfa532202ab1dbe2e1db4671e81d0402281fd9a12" gracePeriod=30 Jul 15 20:27:52.162582 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:52.162553 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:52.374814 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:52.374777 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:53.145890 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:53.145853 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:53.246079 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:53.246047 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:56.688388 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:56.688339 2554 patch_prober.go:28] interesting pod/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 20:27:56.688772 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:56.688447 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" podUID="8b73a338-a660-472b-8b02-ece25a4b953a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 20:27:57.396132 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.396104 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:27:57.480102 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.480059 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b73a338-a660-472b-8b02-ece25a4b953a-istio-token\") pod \"8b73a338-a660-472b-8b02-ece25a4b953a\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " Jul 15 20:27:57.480312 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.480121 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b73a338-a660-472b-8b02-ece25a4b953a-istio-podinfo\") pod \"8b73a338-a660-472b-8b02-ece25a4b953a\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " Jul 15 20:27:57.480312 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.480140 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-istio-data\") pod \"8b73a338-a660-472b-8b02-ece25a4b953a\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " Jul 15 20:27:57.480312 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.480162 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b73a338-a660-472b-8b02-ece25a4b953a-istiod-ca-cert\") pod \"8b73a338-a660-472b-8b02-ece25a4b953a\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " Jul 15 20:27:57.480312 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.480178 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-istio-envoy\") pod \"8b73a338-a660-472b-8b02-ece25a4b953a\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " Jul 15 20:27:57.480312 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.480284 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs2w7\" (UniqueName: \"kubernetes.io/projected/8b73a338-a660-472b-8b02-ece25a4b953a-kube-api-access-hs2w7\") pod \"8b73a338-a660-472b-8b02-ece25a4b953a\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " Jul 15 20:27:57.480588 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.480328 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-credential-socket\") pod \"8b73a338-a660-472b-8b02-ece25a4b953a\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " Jul 15 20:27:57.480588 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.480400 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-workload-certs\") pod \"8b73a338-a660-472b-8b02-ece25a4b953a\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " Jul 15 20:27:57.480588 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.480433 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-workload-socket\") pod \"8b73a338-a660-472b-8b02-ece25a4b953a\" (UID: \"8b73a338-a660-472b-8b02-ece25a4b953a\") " Jul 15 20:27:57.480738 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.480670 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8b73a338-a660-472b-8b02-ece25a4b953a" (UID: "8b73a338-a660-472b-8b02-ece25a4b953a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:27:57.480738 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.480613 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b73a338-a660-472b-8b02-ece25a4b953a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8b73a338-a660-472b-8b02-ece25a4b953a" (UID: "8b73a338-a660-472b-8b02-ece25a4b953a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:27:57.480941 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.480910 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8b73a338-a660-472b-8b02-ece25a4b953a" (UID: "8b73a338-a660-472b-8b02-ece25a4b953a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:27:57.481077 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.480937 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8b73a338-a660-472b-8b02-ece25a4b953a" (UID: "8b73a338-a660-472b-8b02-ece25a4b953a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:27:57.481284 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.481258 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-istio-data" (OuterVolumeSpecName: "istio-data") pod "8b73a338-a660-472b-8b02-ece25a4b953a" (UID: "8b73a338-a660-472b-8b02-ece25a4b953a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:27:57.482643 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.482613 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b73a338-a660-472b-8b02-ece25a4b953a-istio-token" (OuterVolumeSpecName: "istio-token") pod "8b73a338-a660-472b-8b02-ece25a4b953a" (UID: "8b73a338-a660-472b-8b02-ece25a4b953a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:27:57.482643 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.482617 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b73a338-a660-472b-8b02-ece25a4b953a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8b73a338-a660-472b-8b02-ece25a4b953a" (UID: "8b73a338-a660-472b-8b02-ece25a4b953a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:27:57.482643 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.482628 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8b73a338-a660-472b-8b02-ece25a4b953a" (UID: "8b73a338-a660-472b-8b02-ece25a4b953a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:27:57.482872 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.482838 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b73a338-a660-472b-8b02-ece25a4b953a-kube-api-access-hs2w7" (OuterVolumeSpecName: "kube-api-access-hs2w7") pod "8b73a338-a660-472b-8b02-ece25a4b953a" (UID: "8b73a338-a660-472b-8b02-ece25a4b953a"). InnerVolumeSpecName "kube-api-access-hs2w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:27:57.581201 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.581155 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b73a338-a660-472b-8b02-ece25a4b953a-istio-podinfo\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:27:57.581201 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.581197 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-istio-data\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:27:57.581201 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.581211 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b73a338-a660-472b-8b02-ece25a4b953a-istiod-ca-cert\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:27:57.581447 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.581223 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-istio-envoy\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:27:57.581447 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.581236 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs2w7\" (UniqueName: \"kubernetes.io/projected/8b73a338-a660-472b-8b02-ece25a4b953a-kube-api-access-hs2w7\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:27:57.581447 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.581250 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-credential-socket\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:27:57.581447 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.581262 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-workload-certs\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:27:57.581447 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.581275 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a338-a660-472b-8b02-ece25a4b953a-workload-socket\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:27:57.581447 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.581287 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b73a338-a660-472b-8b02-ece25a4b953a-istio-token\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:27:57.981746 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.981710 2554 generic.go:356] "Generic (PLEG): container finished" podID="8b73a338-a660-472b-8b02-ece25a4b953a" containerID="bf92c3273ada452809da409cfa532202ab1dbe2e1db4671e81d0402281fd9a12" exitCode=0 Jul 15 20:27:57.982148 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.981798 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" Jul 15 20:27:57.982148 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.981793 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" event={"ID":"8b73a338-a660-472b-8b02-ece25a4b953a","Type":"ContainerDied","Data":"bf92c3273ada452809da409cfa532202ab1dbe2e1db4671e81d0402281fd9a12"} Jul 15 20:27:57.982148 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.981893 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft" event={"ID":"8b73a338-a660-472b-8b02-ece25a4b953a","Type":"ContainerDied","Data":"ad3a9a16f94435973fbf149d06aac05259687e7d00baa717c41ce1f76451ee50"} Jul 15 20:27:57.982148 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.981925 2554 scope.go:117] "RemoveContainer" containerID="bf92c3273ada452809da409cfa532202ab1dbe2e1db4671e81d0402281fd9a12" Jul 15 20:27:57.992207 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:57.992187 2554 scope.go:117] "RemoveContainer" containerID="62ed9bce1a8740ebc2404925780740873b58a02f30a122833431b3a23706122b" Jul 15 20:27:58.000892 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:58.000873 2554 scope.go:117] "RemoveContainer" containerID="bf92c3273ada452809da409cfa532202ab1dbe2e1db4671e81d0402281fd9a12" Jul 15 20:27:58.001155 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:27:58.001134 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf92c3273ada452809da409cfa532202ab1dbe2e1db4671e81d0402281fd9a12\": container with ID starting with bf92c3273ada452809da409cfa532202ab1dbe2e1db4671e81d0402281fd9a12 not found: ID does not exist" containerID="bf92c3273ada452809da409cfa532202ab1dbe2e1db4671e81d0402281fd9a12" Jul 15 20:27:58.001203 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:58.001167 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf92c3273ada452809da409cfa532202ab1dbe2e1db4671e81d0402281fd9a12"} err="failed to get container status \"bf92c3273ada452809da409cfa532202ab1dbe2e1db4671e81d0402281fd9a12\": rpc error: code = NotFound desc = could not find container \"bf92c3273ada452809da409cfa532202ab1dbe2e1db4671e81d0402281fd9a12\": container with ID starting with bf92c3273ada452809da409cfa532202ab1dbe2e1db4671e81d0402281fd9a12 not found: ID does not exist" Jul 15 20:27:58.001203 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:58.001192 2554 scope.go:117] "RemoveContainer" containerID="62ed9bce1a8740ebc2404925780740873b58a02f30a122833431b3a23706122b" Jul 15 20:27:58.001424 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:27:58.001409 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ed9bce1a8740ebc2404925780740873b58a02f30a122833431b3a23706122b\": container with ID starting with 62ed9bce1a8740ebc2404925780740873b58a02f30a122833431b3a23706122b not found: ID does not exist" containerID="62ed9bce1a8740ebc2404925780740873b58a02f30a122833431b3a23706122b" Jul 15 20:27:58.001464 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:58.001430 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ed9bce1a8740ebc2404925780740873b58a02f30a122833431b3a23706122b"} err="failed to get container status \"62ed9bce1a8740ebc2404925780740873b58a02f30a122833431b3a23706122b\": rpc error: code = NotFound desc = could not find container \"62ed9bce1a8740ebc2404925780740873b58a02f30a122833431b3a23706122b\": container with ID starting with 62ed9bce1a8740ebc2404925780740873b58a02f30a122833431b3a23706122b not found: ID does not exist" Jul 15 20:27:58.006017 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:58.005993 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft"] Jul 15 20:27:58.010796 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:58.010770 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-3ada3b-openshift-default-b9d9df7cf-569ft"] Jul 15 20:27:58.238876 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:27:58.238800 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b73a338-a660-472b-8b02-ece25a4b953a" path="/var/lib/kubelet/pods/8b73a338-a660-472b-8b02-ece25a4b953a/volumes" Jul 15 20:28:02.999561 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:02.999523 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv"] Jul 15 20:28:03.000092 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:02.999792 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" podUID="2a67a619-30ec-4583-96d9-7b80e3febe7f" containerName="maas-api" containerID="cri-o://be8c80b9e71d3f599a816acf2fdcc6e93766a4d6b0a953f05f348280fceaf1b3" gracePeriod=30 Jul 15 20:28:03.044919 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:03.043491 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:03.248044 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:03.248017 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" Jul 15 20:28:03.334229 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:03.334144 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a67a619-30ec-4583-96d9-7b80e3febe7f-maas-api-tls\") pod \"2a67a619-30ec-4583-96d9-7b80e3febe7f\" (UID: \"2a67a619-30ec-4583-96d9-7b80e3febe7f\") " Jul 15 20:28:03.334229 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:03.334204 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a67a619-30ec-4583-96d9-7b80e3febe7f-openshift-service-ca\") pod \"2a67a619-30ec-4583-96d9-7b80e3febe7f\" (UID: \"2a67a619-30ec-4583-96d9-7b80e3febe7f\") " Jul 15 20:28:03.334419 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:03.334264 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xfvk\" (UniqueName: \"kubernetes.io/projected/2a67a619-30ec-4583-96d9-7b80e3febe7f-kube-api-access-5xfvk\") pod \"2a67a619-30ec-4583-96d9-7b80e3febe7f\" (UID: \"2a67a619-30ec-4583-96d9-7b80e3febe7f\") " Jul 15 20:28:03.334615 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:03.334585 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a67a619-30ec-4583-96d9-7b80e3febe7f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2a67a619-30ec-4583-96d9-7b80e3febe7f" (UID: "2a67a619-30ec-4583-96d9-7b80e3febe7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:28:03.336505 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:03.336486 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a67a619-30ec-4583-96d9-7b80e3febe7f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2a67a619-30ec-4583-96d9-7b80e3febe7f" (UID: "2a67a619-30ec-4583-96d9-7b80e3febe7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:28:03.336572 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:03.336506 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a67a619-30ec-4583-96d9-7b80e3febe7f-kube-api-access-5xfvk" (OuterVolumeSpecName: "kube-api-access-5xfvk") pod "2a67a619-30ec-4583-96d9-7b80e3febe7f" (UID: "2a67a619-30ec-4583-96d9-7b80e3febe7f"). InnerVolumeSpecName "kube-api-access-5xfvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:28:03.435109 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:03.435071 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a67a619-30ec-4583-96d9-7b80e3febe7f-maas-api-tls\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:28:03.435109 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:03.435103 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a67a619-30ec-4583-96d9-7b80e3febe7f-openshift-service-ca\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:28:03.435109 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:03.435113 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xfvk\" (UniqueName: \"kubernetes.io/projected/2a67a619-30ec-4583-96d9-7b80e3febe7f-kube-api-access-5xfvk\") on node \"ip-10-0-135-139.ec2.internal\" DevicePath \"\"" Jul 15 20:28:04.019548 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:04.019500 2554 generic.go:356] "Generic (PLEG): container finished" podID="2a67a619-30ec-4583-96d9-7b80e3febe7f" containerID="be8c80b9e71d3f599a816acf2fdcc6e93766a4d6b0a953f05f348280fceaf1b3" exitCode=0 Jul 15 20:28:04.019548 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:04.019548 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" Jul 15 20:28:04.020048 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:04.019588 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" event={"ID":"2a67a619-30ec-4583-96d9-7b80e3febe7f","Type":"ContainerDied","Data":"be8c80b9e71d3f599a816acf2fdcc6e93766a4d6b0a953f05f348280fceaf1b3"} Jul 15 20:28:04.020048 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:04.019649 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv" event={"ID":"2a67a619-30ec-4583-96d9-7b80e3febe7f","Type":"ContainerDied","Data":"74e681d973b362d77d9b73a5481c29e183585f6f8ce8ad96865ba0a623636226"} Jul 15 20:28:04.020048 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:04.019678 2554 scope.go:117] "RemoveContainer" containerID="be8c80b9e71d3f599a816acf2fdcc6e93766a4d6b0a953f05f348280fceaf1b3" Jul 15 20:28:04.029548 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:04.029527 2554 scope.go:117] "RemoveContainer" containerID="be8c80b9e71d3f599a816acf2fdcc6e93766a4d6b0a953f05f348280fceaf1b3" Jul 15 20:28:04.029893 ip-10-0-135-139 kubenswrapper[2554]: E0715 20:28:04.029865 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be8c80b9e71d3f599a816acf2fdcc6e93766a4d6b0a953f05f348280fceaf1b3\": container with ID starting with be8c80b9e71d3f599a816acf2fdcc6e93766a4d6b0a953f05f348280fceaf1b3 not found: ID does not exist" containerID="be8c80b9e71d3f599a816acf2fdcc6e93766a4d6b0a953f05f348280fceaf1b3" Jul 15 20:28:04.029982 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:04.029904 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8c80b9e71d3f599a816acf2fdcc6e93766a4d6b0a953f05f348280fceaf1b3"} err="failed to get container status \"be8c80b9e71d3f599a816acf2fdcc6e93766a4d6b0a953f05f348280fceaf1b3\": rpc error: code = NotFound desc = could not find container \"be8c80b9e71d3f599a816acf2fdcc6e93766a4d6b0a953f05f348280fceaf1b3\": container with ID starting with be8c80b9e71d3f599a816acf2fdcc6e93766a4d6b0a953f05f348280fceaf1b3 not found: ID does not exist" Jul 15 20:28:04.044591 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:04.044559 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv"] Jul 15 20:28:04.049449 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:04.049425 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-54098e-6997fc4668-mshbv"] Jul 15 20:28:04.239787 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:04.239733 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a67a619-30ec-4583-96d9-7b80e3febe7f" path="/var/lib/kubelet/pods/2a67a619-30ec-4583-96d9-7b80e3febe7f/volumes" Jul 15 20:28:06.142885 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:06.142853 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ggt9w"] Jul 15 20:28:06.508429 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:06.508400 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 20:28:06.512893 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:06.512865 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 20:28:06.519199 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:06.519176 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 20:28:06.523378 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:06.523360 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxd2f_16c0161e-1bfd-4b7b-93e0-a0da2acad94f/ovn-acl-logging/0.log" Jul 15 20:28:07.610981 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:07.610951 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:13.586128 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:13.586099 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:13.687917 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:13.687877 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:18.277824 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:18.277770 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:18.486658 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:18.486607 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:24.798615 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:24.798586 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c8f5fdd94-bh52s_ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c/copy-manifests/0.log" Jul 15 20:28:24.806465 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:24.806422 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c8f5fdd94-bh52s_ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c/manager/0.log" Jul 15 20:28:25.613344 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:25.613311 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29735760-kvmlk_f7d95f85-4de7-4369-8881-4c281515cbb4/cleanup/0.log" Jul 15 20:28:25.723499 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:25.723464 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29735775-qp6w4_e14970f6-54c0-4be5-bc2a-3a6f253214e3/cleanup/0.log" Jul 15 20:28:25.837937 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:25.837905 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-10eed66e-9677l_aeaf38aa-db6e-45be-bb8b-44b65c707670/revoke-keys/0.log" Jul 15 20:28:25.950402 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:25.950376 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-7a4fe348-hbdnj_0dd01ef1-c291-4c45-8537-bc3e6a03759e/revoke-keys/0.log" Jul 15 20:28:26.172475 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:26.172450 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-772fb7-mlhr2_d28d4f7e-710c-4409-88c1-ad1610a7d930/revoke-keys/0.log" Jul 15 20:28:26.283743 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:26.283653 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-60ba2b89-2m9v9_a71b1f9c-7312-4760-b9c6-a496525a655a/revoke-keys/0.log" Jul 15 20:28:26.399027 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:26.398991 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-c923f1-4sv95_d25d107e-24f5-4953-aad4-1027fe5d1f95/revoke-keys/0.log" Jul 15 20:28:26.514841 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:26.514815 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-a27bd5-bmttd_2efc45e0-e8fb-489f-91f1-f3c2ed055f03/revoke-keys/0.log" Jul 15 20:28:26.630462 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:26.630383 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-73e5b6fd-kfbrk_85a9717a-2f91-48a3-b207-4204901a8e69/revoke-keys/0.log" Jul 15 20:28:26.745686 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:26.745658 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-82e04801-rmlnh_d011e8df-2238-4396-b9fe-057a915f2254/revoke-keys/0.log" Jul 15 20:28:26.860991 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:26.860961 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-8950fff5-6j4dw_331d99af-d49b-41a8-bf24-3ec0b87b3145/revoke-keys/0.log" Jul 15 20:28:28.045083 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:28.045048 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs_4a77b95a-586b-42ee-bb28-7125e1df08ca/util/0.log" Jul 15 20:28:28.051190 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:28.051168 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs_4a77b95a-586b-42ee-bb28-7125e1df08ca/pull/0.log" Jul 15 20:28:28.056611 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:28.056582 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs_4a77b95a-586b-42ee-bb28-7125e1df08ca/extract/0.log" Jul 15 20:28:28.173714 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:28.173690 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr_5d0a135d-3c27-4f90-9b18-1d40028588cd/util/0.log" Jul 15 20:28:28.180294 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:28.180268 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr_5d0a135d-3c27-4f90-9b18-1d40028588cd/pull/0.log" Jul 15 20:28:28.186465 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:28.186448 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr_5d0a135d-3c27-4f90-9b18-1d40028588cd/extract/0.log" Jul 15 20:28:28.297207 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:28.297123 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws_e9ce92c8-b704-4173-8fd5-20b926a4d18b/util/0.log" Jul 15 20:28:28.303200 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:28.303178 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws_e9ce92c8-b704-4173-8fd5-20b926a4d18b/pull/0.log" Jul 15 20:28:28.309401 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:28.309374 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws_e9ce92c8-b704-4173-8fd5-20b926a4d18b/extract/0.log" Jul 15 20:28:28.425432 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:28.425399 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2_28945aeb-cc94-41c9-a5ff-9ff37423e55b/util/0.log" Jul 15 20:28:28.431649 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:28.431626 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2_28945aeb-cc94-41c9-a5ff-9ff37423e55b/pull/0.log" Jul 15 20:28:28.437337 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:28.437311 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2_28945aeb-cc94-41c9-a5ff-9ff37423e55b/extract/0.log" Jul 15 20:28:29.238072 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:29.238044 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ggt9w_e5b93a46-4b97-48ea-861c-e6ca6a9ffdef/limitador/0.log" Jul 15 20:28:29.847614 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:29.847587 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-8nxrb_d49f4e21-2e82-48b2-ab85-9d2dae1a6241/discovery/0.log" Jul 15 20:28:30.540072 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:30.540044 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-656f8d6d5b-5lzhw_eb9ebe84-37a6-4f73-880e-dad90c6453c7/router/0.log" Jul 15 20:28:30.994636 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:30.994609 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-jxqkv_c6f95134-bc21-41b6-870d-980b333968ba/main/0.log" Jul 15 20:28:37.288473 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:37.288444 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:37.387656 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:37.387626 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:40.534016 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:40.533990 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rv9rh_87560cf2-a9d6-46e4-86dd-eac6f20cc8c8/global-pull-secret-syncer/0.log" Jul 15 20:28:40.656955 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:40.656920 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" Jul 15 20:28:40.788213 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:40.788132 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-139.ec2.internal_4dd0763a6110d8ba2bd7fcafeff6d0d7/haproxy/0.log" Jul 15 20:28:41.056318 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:41.056233 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:41.156914 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:41.156877 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:45.270894 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:45.270858 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs_4a77b95a-586b-42ee-bb28-7125e1df08ca/extract/0.log" Jul 15 20:28:45.309060 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:45.309026 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs_4a77b95a-586b-42ee-bb28-7125e1df08ca/util/0.log" Jul 15 20:28:45.332291 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:45.332252 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n6rzs_4a77b95a-586b-42ee-bb28-7125e1df08ca/pull/0.log" Jul 15 20:28:45.377370 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:45.377344 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr_5d0a135d-3c27-4f90-9b18-1d40028588cd/extract/0.log" Jul 15 20:28:45.418898 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:45.418864 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr_5d0a135d-3c27-4f90-9b18-1d40028588cd/util/0.log" Jul 15 20:28:45.461100 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:45.461078 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5lwr_5d0a135d-3c27-4f90-9b18-1d40028588cd/pull/0.log" Jul 15 20:28:45.497068 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:45.497036 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws_e9ce92c8-b704-4173-8fd5-20b926a4d18b/extract/0.log" Jul 15 20:28:45.535722 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:45.535645 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws_e9ce92c8-b704-4173-8fd5-20b926a4d18b/util/0.log" Jul 15 20:28:45.561649 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:45.561622 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pd2ws_e9ce92c8-b704-4173-8fd5-20b926a4d18b/pull/0.log" Jul 15 20:28:45.602637 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:45.602612 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2_28945aeb-cc94-41c9-a5ff-9ff37423e55b/extract/0.log" Jul 15 20:28:45.625989 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:45.625961 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2_28945aeb-cc94-41c9-a5ff-9ff37423e55b/util/0.log" Jul 15 20:28:45.646326 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:45.646303 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15brh2_28945aeb-cc94-41c9-a5ff-9ff37423e55b/pull/0.log" Jul 15 20:28:46.423439 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:46.423411 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ggt9w_e5b93a46-4b97-48ea-861c-e6ca6a9ffdef/limitador/0.log" Jul 15 20:28:48.403098 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:48.403068 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-fksq2_88a9eb45-3875-4d7f-aab1-e1aeb90333d6/cluster-monitoring-operator/0.log" Jul 15 20:28:48.579917 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:48.579888 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7nqwk_caa0af65-db73-412a-9770-c8d23a985201/node-exporter/0.log" Jul 15 20:28:48.601543 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:48.601508 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7nqwk_caa0af65-db73-412a-9770-c8d23a985201/kube-rbac-proxy/0.log" Jul 15 20:28:48.622891 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:48.622865 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7nqwk_caa0af65-db73-412a-9770-c8d23a985201/init-textfile/0.log" Jul 15 20:28:50.280428 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.280393 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns"] Jul 15 20:28:50.280938 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.280770 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b73a338-a660-472b-8b02-ece25a4b953a" containerName="istio-proxy" Jul 15 20:28:50.280938 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.280794 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b73a338-a660-472b-8b02-ece25a4b953a" containerName="istio-proxy" Jul 15 20:28:50.280938 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.280807 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1168f03a-3a28-4e50-8020-c53b1fb33bb9" containerName="maas-api" Jul 15 20:28:50.280938 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.280814 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1168f03a-3a28-4e50-8020-c53b1fb33bb9" containerName="maas-api" Jul 15 20:28:50.280938 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.280823 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a67a619-30ec-4583-96d9-7b80e3febe7f" containerName="maas-api" Jul 15 20:28:50.280938 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.280828 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a67a619-30ec-4583-96d9-7b80e3febe7f" containerName="maas-api" Jul 15 20:28:50.280938 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.280839 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b73a338-a660-472b-8b02-ece25a4b953a" containerName="istio-proxy" Jul 15 20:28:50.280938 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.280844 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b73a338-a660-472b-8b02-ece25a4b953a" containerName="istio-proxy" Jul 15 20:28:50.280938 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.280915 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1168f03a-3a28-4e50-8020-c53b1fb33bb9" containerName="maas-api" Jul 15 20:28:50.280938 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.280926 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a67a619-30ec-4583-96d9-7b80e3febe7f" containerName="maas-api" Jul 15 20:28:50.280938 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.280932 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b73a338-a660-472b-8b02-ece25a4b953a" containerName="istio-proxy" Jul 15 20:28:50.280938 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.280938 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b73a338-a660-472b-8b02-ece25a4b953a" containerName="istio-proxy" Jul 15 20:28:50.283965 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.283943 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" Jul 15 20:28:50.286481 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.286455 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8sl5g\"/\"openshift-service-ca.crt\"" Jul 15 20:28:50.286481 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.286467 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8sl5g\"/\"kube-root-ca.crt\"" Jul 15 20:28:50.287274 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.287248 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8sl5g\"/\"default-dockercfg-8lcv6\"" Jul 15 20:28:50.291726 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.291701 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns"] Jul 15 20:28:50.334144 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.334099 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk7wm\" (UniqueName: \"kubernetes.io/projected/22501557-7f8d-4971-830d-3d25e81f9a90-kube-api-access-pk7wm\") pod \"perf-node-gather-daemonset-r6hns\" (UID: \"22501557-7f8d-4971-830d-3d25e81f9a90\") " pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" Jul 15 20:28:50.334144 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.334142 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22501557-7f8d-4971-830d-3d25e81f9a90-proc\") pod \"perf-node-gather-daemonset-r6hns\" (UID: \"22501557-7f8d-4971-830d-3d25e81f9a90\") " pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" Jul 15 20:28:50.337147 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.334172 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22501557-7f8d-4971-830d-3d25e81f9a90-lib-modules\") pod \"perf-node-gather-daemonset-r6hns\" (UID: \"22501557-7f8d-4971-830d-3d25e81f9a90\") " pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" Jul 15 20:28:50.337147 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.334226 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22501557-7f8d-4971-830d-3d25e81f9a90-podres\") pod \"perf-node-gather-daemonset-r6hns\" (UID: \"22501557-7f8d-4971-830d-3d25e81f9a90\") " pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" Jul 15 20:28:50.337147 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.334271 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22501557-7f8d-4971-830d-3d25e81f9a90-sys\") pod \"perf-node-gather-daemonset-r6hns\" (UID: \"22501557-7f8d-4971-830d-3d25e81f9a90\") " pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" Jul 15 20:28:50.434698 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.434655 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22501557-7f8d-4971-830d-3d25e81f9a90-sys\") pod \"perf-node-gather-daemonset-r6hns\" (UID: \"22501557-7f8d-4971-830d-3d25e81f9a90\") " pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" Jul 15 20:28:50.434886 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.434804 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk7wm\" (UniqueName: \"kubernetes.io/projected/22501557-7f8d-4971-830d-3d25e81f9a90-kube-api-access-pk7wm\") pod \"perf-node-gather-daemonset-r6hns\" (UID: \"22501557-7f8d-4971-830d-3d25e81f9a90\") " pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" Jul 15 20:28:50.434886 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.434817 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22501557-7f8d-4971-830d-3d25e81f9a90-sys\") pod \"perf-node-gather-daemonset-r6hns\" (UID: \"22501557-7f8d-4971-830d-3d25e81f9a90\") " pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" Jul 15 20:28:50.434886 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.434835 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22501557-7f8d-4971-830d-3d25e81f9a90-proc\") pod \"perf-node-gather-daemonset-r6hns\" (UID: \"22501557-7f8d-4971-830d-3d25e81f9a90\") " pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" Jul 15 20:28:50.434886 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.434874 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22501557-7f8d-4971-830d-3d25e81f9a90-lib-modules\") pod \"perf-node-gather-daemonset-r6hns\" (UID: \"22501557-7f8d-4971-830d-3d25e81f9a90\") " pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" Jul 15 20:28:50.435104 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.434895 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22501557-7f8d-4971-830d-3d25e81f9a90-proc\") pod \"perf-node-gather-daemonset-r6hns\" (UID: \"22501557-7f8d-4971-830d-3d25e81f9a90\") " pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" Jul 15 20:28:50.435104 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.434902 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22501557-7f8d-4971-830d-3d25e81f9a90-podres\") pod \"perf-node-gather-daemonset-r6hns\" (UID: \"22501557-7f8d-4971-830d-3d25e81f9a90\") " pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" Jul 15 20:28:50.435104 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.435006 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22501557-7f8d-4971-830d-3d25e81f9a90-podres\") pod \"perf-node-gather-daemonset-r6hns\" (UID: \"22501557-7f8d-4971-830d-3d25e81f9a90\") " pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" Jul 15 20:28:50.435104 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.435021 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22501557-7f8d-4971-830d-3d25e81f9a90-lib-modules\") pod \"perf-node-gather-daemonset-r6hns\" (UID: \"22501557-7f8d-4971-830d-3d25e81f9a90\") " pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" Jul 15 20:28:50.443595 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.443574 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk7wm\" (UniqueName: \"kubernetes.io/projected/22501557-7f8d-4971-830d-3d25e81f9a90-kube-api-access-pk7wm\") pod \"perf-node-gather-daemonset-r6hns\" (UID: \"22501557-7f8d-4971-830d-3d25e81f9a90\") " pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" Jul 15 20:28:50.517375 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.517348 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-k8brr_7a0eeb16-5201-43e8-a850-a7a3cdedc3a1/networking-console-plugin/0.log" Jul 15 20:28:50.596536 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.596434 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" Jul 15 20:28:50.739673 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:50.739645 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns"] Jul 15 20:28:50.740952 ip-10-0-135-139 kubenswrapper[2554]: W0715 20:28:50.740926 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod22501557_7f8d_4971_830d_3d25e81f9a90.slice/crio-5971c8bb644db53eb2a950a605957896c180c937c2a288e77269ed08986a13c5 WatchSource:0}: Error finding container 5971c8bb644db53eb2a950a605957896c180c937c2a288e77269ed08986a13c5: Status 404 returned error can't find the container with id 5971c8bb644db53eb2a950a605957896c180c937c2a288e77269ed08986a13c5 Jul 15 20:28:51.159327 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:51.159297 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/1.log" Jul 15 20:28:51.163540 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:51.163519 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2ldzn_b21f8771-ffd5-4487-812b-bfbaa7718461/console-operator/2.log" Jul 15 20:28:51.289486 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:51.289447 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" event={"ID":"22501557-7f8d-4971-830d-3d25e81f9a90","Type":"ContainerStarted","Data":"ff23f9590c3b615ae411d635167ede5cf1d96c472b150f75a023c6a17da367a1"} Jul 15 20:28:51.289978 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:51.289505 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" event={"ID":"22501557-7f8d-4971-830d-3d25e81f9a90","Type":"ContainerStarted","Data":"5971c8bb644db53eb2a950a605957896c180c937c2a288e77269ed08986a13c5"} Jul 15 20:28:51.289978 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:51.289533 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" Jul 15 20:28:51.308487 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:51.308437 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" podStartSLOduration=1.308422212 podStartE2EDuration="1.308422212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 20:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:28:51.306155186 +0000 UTC m=+3045.550661647" watchObservedRunningTime="2026-07-15 20:28:51.308422212 +0000 UTC m=+3045.552928672" Jul 15 20:28:52.222194 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:52.222163 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-zms8d_99bcfa5f-a482-4e96-bc3f-a6e3365078d3/volume-data-source-validator/0.log" Jul 15 20:28:53.363947 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:53.363916 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zgrjv_86437276-af54-49be-acf6-91f82d55e92c/dns/0.log" Jul 15 20:28:53.385309 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:53.385285 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zgrjv_86437276-af54-49be-acf6-91f82d55e92c/kube-rbac-proxy/0.log" Jul 15 20:28:53.476901 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:53.476871 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zswzk_67b2bcf5-c147-4a76-b620-b0cec94a0132/dns-node-resolver/0.log" Jul 15 20:28:54.127612 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:54.127581 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wxt6l_2d86b57f-2938-41d5-8d6e-d2a17177030d/node-ca/0.log" Jul 15 20:28:55.222581 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:55.222555 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-8nxrb_d49f4e21-2e82-48b2-ab85-9d2dae1a6241/discovery/0.log" Jul 15 20:28:55.434676 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:55.434645 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-656f8d6d5b-5lzhw_eb9ebe84-37a6-4f73-880e-dad90c6453c7/router/0.log" Jul 15 20:28:56.123833 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:56.123801 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2288d_d896f9b8-775c-4130-820c-59831ff12641/serve-healthcheck-canary/0.log" Jul 15 20:28:56.668477 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:56.668439 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-pmncb_e2691146-bc04-4c7b-bb0c-b3377762da6e/insights-operator/0.log" Jul 15 20:28:56.669184 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:56.669168 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-pmncb_e2691146-bc04-4c7b-bb0c-b3377762da6e/insights-operator/1.log" Jul 15 20:28:56.690112 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:56.690067 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5rb4w_38c84af0-f6d3-4217-ba8a-87fb6912c97e/kube-rbac-proxy/0.log" Jul 15 20:28:56.712213 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:56.712191 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5rb4w_38c84af0-f6d3-4217-ba8a-87fb6912c97e/exporter/0.log" Jul 15 20:28:56.737135 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:56.737112 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5rb4w_38c84af0-f6d3-4217-ba8a-87fb6912c97e/extractor/0.log" Jul 15 20:28:57.305396 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:57.305370 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8sl5g/perf-node-gather-daemonset-r6hns" Jul 15 20:28:59.692941 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:59.692907 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c8f5fdd94-bh52s_ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c/manager/0.log" Jul 15 20:28:59.738312 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:28:59.738287 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c8f5fdd94-bh52s_ffc9f9e0-bb94-40f7-91de-2a91d5ed4e4c/copy-manifests/0.log" Jul 15 20:29:01.056291 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:29:01.056206 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-8nxrb_d49f4e21-2e82-48b2-ab85-9d2dae1a6241/discovery/0.log" Jul 15 20:29:01.197410 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:29:01.197375 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-656f8d6d5b-5lzhw_eb9ebe84-37a6-4f73-880e-dad90c6453c7/router/0.log" Jul 15 20:29:01.341361 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:29:01.341276 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c89ffd76c-hxx48_93279220-48fc-4ec2-b2ac-14b2d6abf2a6/manager/0.log" Jul 15 20:29:01.409663 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:29:01.409637 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-kdxdb_2d0a0425-17b3-4b5c-8cf1-a73b0ead7507/openshift-lws-operator/0.log" Jul 15 20:29:02.560661 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:29:02.560627 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-8nxrb_d49f4e21-2e82-48b2-ab85-9d2dae1a6241/discovery/0.log" Jul 15 20:29:02.703668 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:29:02.703633 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-656f8d6d5b-5lzhw_eb9ebe84-37a6-4f73-880e-dad90c6453c7/router/0.log" Jul 15 20:29:03.377652 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:29:03.377616 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:29:03.452521 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:29:03.452487 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mhtt6_1a81be65-7d76-41b7-962a-c0380b732b1d/konnectivity-agent/0.log" limit=1024 Jul 15 20:29:06.456578 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:29:06.456552 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-87gx7_bd5cd4f0-d31d-4f1f-877d-53537b0fc122/migrator/0.log" Jul 15 20:29:06.501718 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:29:06.501692 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-87gx7_bd5cd4f0-d31d-4f1f-877d-53537b0fc122/graceful-termination/0.log" Jul 15 20:29:06.906765 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:29:06.906657 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-69x22_2ae8e73f-008e-4977-8153-342f4a7460a5/kube-storage-version-migrator-operator/1.log" Jul 15 20:29:06.907451 ip-10-0-135-139 kubenswrapper[2554]: I0715 20:29:06.907429 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-69x22_2ae8e73f-008e-4977-8153-342f4a7460a5/kube-storage-version-migrator-operator/0.log"