Jul 13 21:58:06.065407 ip-10-0-137-183 systemd[1]: Starting Kubernetes Kubelet... Jul 13 21:58:06.525037 ip-10-0-137-183 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 21:58:06.525037 ip-10-0-137-183 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 13 21:58:06.525037 ip-10-0-137-183 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 21:58:06.525037 ip-10-0-137-183 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 13 21:58:06.525037 ip-10-0-137-183 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 21:58:06.525037 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.518270 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 13 21:58:06.525037 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523714 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 21:58:06.525037 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523726 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 21:58:06.525037 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523730 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 21:58:06.525037 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523736 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 21:58:06.525037 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523740 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 21:58:06.525037 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523744 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 21:58:06.525037 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523747 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 21:58:06.525037 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523749 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523752 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523755 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523757 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523760 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523762 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523770 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523773 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523776 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523778 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523781 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523784 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523786 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523789 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523792 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523795 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523797 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523800 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523802 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523805 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 21:58:06.527655 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523808 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 21:58:06.545617 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523810 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 21:58:06.545617 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523813 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 21:58:06.545617 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523816 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 21:58:06.545617 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523818 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 21:58:06.545617 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523821 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 21:58:06.545617 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523824 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 21:58:06.545617 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523826 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 21:58:06.545617 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523829 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 21:58:06.545617 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523832 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 21:58:06.545617 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523835 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 21:58:06.545617 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523837 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 21:58:06.545617 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523841 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 21:58:06.545617 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523845 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 21:58:06.545617 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523848 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 21:58:06.545617 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523851 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 21:58:06.545617 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523866 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 21:58:06.545617 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523870 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 21:58:06.545617 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523874 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 21:58:06.545617 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523878 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523881 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523884 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523887 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523889 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523892 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523895 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523898 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523901 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523903 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523906 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523916 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523920 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523923 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523926 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523929 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523931 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523934 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523937 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523939 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 21:58:06.549237 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523942 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523946 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523948 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523951 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523953 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523956 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523958 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523960 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523963 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523965 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523968 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523971 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523973 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523981 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523984 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523987 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523990 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523992 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523995 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.523997 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 21:58:06.563774 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524001 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524004 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524006 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524009 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524011 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524014 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524017 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524019 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524023 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524026 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524029 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524032 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524035 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524764 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524771 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524774 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524777 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524780 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524783 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524786 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 21:58:06.567809 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524788 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524792 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524794 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524797 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524799 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524802 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524805 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524808 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524811 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524813 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524816 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524818 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524821 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524824 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524827 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524829 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524832 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524835 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524837 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524840 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 21:58:06.581361 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524842 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 21:58:06.585074 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524845 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 21:58:06.585074 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524847 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 21:58:06.585074 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524851 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 21:58:06.585074 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524853 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 21:58:06.585074 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524856 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 21:58:06.585074 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524858 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 21:58:06.585074 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524861 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 21:58:06.585074 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524863 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 21:58:06.585074 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524879 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 21:58:06.585074 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524882 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 21:58:06.585074 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524885 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 21:58:06.585074 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524888 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 21:58:06.585074 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524891 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 21:58:06.585074 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524893 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 21:58:06.585074 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524896 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 21:58:06.585074 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524898 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 21:58:06.585074 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524901 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 21:58:06.585074 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524903 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 21:58:06.585074 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524907 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 21:58:06.587217 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524909 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 21:58:06.587217 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524912 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 21:58:06.587217 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524915 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 21:58:06.587217 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524918 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 21:58:06.587217 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524921 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 21:58:06.587217 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524923 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 21:58:06.587217 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524926 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 21:58:06.587217 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524929 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 21:58:06.587217 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524931 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 21:58:06.587217 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524934 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 21:58:06.587217 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524936 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 21:58:06.587217 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524939 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 21:58:06.587217 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524942 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 21:58:06.587217 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524945 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 21:58:06.587217 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524948 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 21:58:06.587217 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524950 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 21:58:06.587217 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524955 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 21:58:06.587217 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524958 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 21:58:06.587217 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524961 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524964 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524967 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524969 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524972 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524974 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524977 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524980 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524983 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524985 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524988 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524990 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524993 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524995 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.524998 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525001 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525005 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525008 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525011 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525015 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 21:58:06.591147 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525018 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 21:58:06.593596 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525021 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 21:58:06.593596 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525025 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 21:58:06.593596 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525029 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 21:58:06.593596 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525032 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 21:58:06.593596 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525034 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 13 21:58:06.593596 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525037 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 21:58:06.593596 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525040 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 21:58:06.593596 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525043 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 21:58:06.593596 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525045 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 21:58:06.593596 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525060 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 21:58:06.593596 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525063 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 21:58:06.593596 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525066 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 21:58:06.593596 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525068 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 21:58:06.593596 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525479 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 21:58:06.593596 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525485 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 21:58:06.593596 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525490 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 21:58:06.593596 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525494 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 21:58:06.593596 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525497 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 21:58:06.593596 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525500 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 21:58:06.597535 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525503 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 21:58:06.597535 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525506 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 21:58:06.597535 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525509 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 21:58:06.597535 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525512 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 21:58:06.597535 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525514 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 21:58:06.597535 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525517 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 21:58:06.597535 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525525 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 21:58:06.597535 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525528 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 21:58:06.597535 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525530 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 21:58:06.597535 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525533 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 21:58:06.597535 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525536 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 21:58:06.597535 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525538 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 21:58:06.597535 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525541 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 21:58:06.597535 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525543 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 21:58:06.597535 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525546 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 21:58:06.597535 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525548 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 21:58:06.597535 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525551 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 21:58:06.597535 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525553 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 21:58:06.597535 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525556 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525558 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525561 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525563 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525566 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525569 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525572 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525575 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525577 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525581 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525583 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525585 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525588 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525590 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525593 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525595 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525598 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525601 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525603 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525606 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 21:58:06.599909 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525608 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 13 21:58:06.599483 ip-10-0-137-183 systemd[1]: Started Kubernetes Kubelet. Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525611 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525614 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525616 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525619 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525621 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525624 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525626 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525629 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525631 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525634 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525637 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525639 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525642 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525644 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525647 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525649 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525652 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525655 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525657 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 21:58:06.605081 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525661 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525664 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525667 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525669 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525672 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525674 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525677 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525680 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525682 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525685 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525688 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525690 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525693 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525695 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525699 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525702 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525705 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525709 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525711 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525714 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 21:58:06.608866 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525717 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525720 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525723 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525727 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525731 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525734 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525736 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525739 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525742 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525745 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525747 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525750 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525753 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525755 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.525758 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525829 2569 flags.go:64] FLAG: --address="0.0.0.0" Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525836 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525844 2569 flags.go:64] FLAG: --anonymous-auth="true" Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525851 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525861 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525866 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 13 21:58:06.614314 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525875 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525891 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525894 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525897 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525901 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525904 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525913 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525917 2569 flags.go:64] FLAG: --cgroup-root="" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525920 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525922 2569 flags.go:64] FLAG: --client-ca-file="" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525925 2569 flags.go:64] FLAG: --cloud-provider="external" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525928 2569 flags.go:64] FLAG: --cluster-dns="[]" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525935 2569 flags.go:64] FLAG: --cluster-domain="" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525938 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525941 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525944 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525947 2569 flags.go:64] FLAG: --container-log-max-files="5" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525952 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525955 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525958 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525962 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525965 2569 flags.go:64] FLAG: --contention-profiling="false" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525968 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525971 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 13 21:58:06.617571 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525977 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525980 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525985 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525988 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525990 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525994 2569 flags.go:64] FLAG: --enable-load-reader="false" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.525997 2569 flags.go:64] FLAG: --enable-server="true" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526000 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526007 2569 flags.go:64] FLAG: --event-burst="100" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526010 2569 flags.go:64] FLAG: --event-qps="50" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526013 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526016 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526019 2569 flags.go:64] FLAG: --eviction-hard="" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526022 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526025 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526034 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526037 2569 flags.go:64] FLAG: --eviction-soft="" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526040 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526043 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526046 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526062 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526065 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526068 2569 flags.go:64] FLAG: --fail-swap-on="true" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526072 2569 flags.go:64] FLAG: --feature-gates="" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526075 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jul 13 21:58:06.626810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526078 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526081 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526084 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526089 2569 flags.go:64] FLAG: --healthz-port="10248" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526093 2569 flags.go:64] FLAG: --help="false" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526096 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-137-183.ec2.internal" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526099 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526104 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526107 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526110 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526113 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526116 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526119 2569 flags.go:64] FLAG: --image-service-endpoint="" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526122 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526125 2569 flags.go:64] FLAG: --kube-api-burst="100" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526128 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526131 2569 flags.go:64] FLAG: --kube-api-qps="50" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526134 2569 flags.go:64] FLAG: --kube-reserved="" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526137 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526140 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526143 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526146 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526149 2569 flags.go:64] FLAG: --lock-file="" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526157 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 13 21:58:06.790527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526160 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526163 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526168 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526171 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526174 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526177 2569 flags.go:64] FLAG: --logging-format="text" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526180 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526183 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526186 2569 flags.go:64] FLAG: --manifest-url="" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526188 2569 flags.go:64] FLAG: --manifest-url-header="" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526192 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526197 2569 flags.go:64] FLAG: --max-open-files="1000000" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526201 2569 flags.go:64] FLAG: --max-pods="110" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526204 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526207 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526212 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526215 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526218 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526221 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526224 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526231 2569 flags.go:64] FLAG: --node-status-max-images="50" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526234 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526237 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jul 13 21:58:06.809964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526240 2569 flags.go:64] FLAG: --pod-cidr="" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526243 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526248 2569 flags.go:64] FLAG: --pod-manifest-path="" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526250 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526254 2569 flags.go:64] FLAG: --pods-per-core="0" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526257 2569 flags.go:64] FLAG: --port="10250" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526260 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526263 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03dee6f4cc68a700a" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526266 2569 flags.go:64] FLAG: --qos-reserved="" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526269 2569 flags.go:64] FLAG: --read-only-port="10255" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526272 2569 flags.go:64] FLAG: --register-node="true" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526275 2569 flags.go:64] FLAG: --register-with-taints="" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526279 2569 flags.go:64] FLAG: --registry-burst="10" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526281 2569 flags.go:64] FLAG: --registry-qps="5" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526284 2569 flags.go:64] FLAG: --reserved-cpus="" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526287 2569 flags.go:64] FLAG: --reserved-memory="" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526291 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526293 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526296 2569 flags.go:64] FLAG: --rotate-certificates="false" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526299 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526302 2569 flags.go:64] FLAG: --runonce="false" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526306 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526309 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526312 2569 flags.go:64] FLAG: --seccomp-default="false" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526316 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 13 21:58:06.815375 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526320 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526323 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526326 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526329 2569 flags.go:64] FLAG: --storage-driver-password="root" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526332 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526335 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526338 2569 flags.go:64] FLAG: --storage-driver-user="root" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526340 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526343 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526346 2569 flags.go:64] FLAG: --system-cgroups="" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526349 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526354 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526357 2569 flags.go:64] FLAG: --tls-cert-file="" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526360 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526364 2569 flags.go:64] FLAG: --tls-min-version="" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526367 2569 flags.go:64] FLAG: --tls-private-key-file="" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526370 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526373 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526376 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526379 2569 flags.go:64] FLAG: --v="2" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526383 2569 flags.go:64] FLAG: --version="false" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526387 2569 flags.go:64] FLAG: --vmodule="" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526390 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.526394 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526512 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 21:58:06.816213 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526518 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526523 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526528 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526535 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526539 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526543 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526549 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526553 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526558 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526562 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526566 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526571 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526574 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526577 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526579 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526582 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526585 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526588 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526590 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526593 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 21:58:06.817692 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526595 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 21:58:06.818438 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526598 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 21:58:06.818438 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526601 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 21:58:06.818438 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526603 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 21:58:06.818438 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526606 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 21:58:06.818438 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526609 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 21:58:06.818438 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526612 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 21:58:06.818438 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526614 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 21:58:06.818438 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526617 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 21:58:06.818438 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526619 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 21:58:06.818438 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526622 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 21:58:06.818438 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526626 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 21:58:06.818438 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526629 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 21:58:06.818438 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526632 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 21:58:06.818438 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526634 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 21:58:06.818438 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526638 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 21:58:06.818438 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526641 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 21:58:06.818438 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526643 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 21:58:06.818438 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526648 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 21:58:06.818438 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526651 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526653 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526656 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526658 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526661 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526664 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526666 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526669 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526672 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526674 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526677 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526680 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526682 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526685 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526687 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526690 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526692 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526695 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526698 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526704 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 21:58:06.819175 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526707 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526709 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526712 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526714 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526717 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526719 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526722 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526725 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526729 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526731 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526734 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526738 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526740 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526743 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526746 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526748 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526751 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526754 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526757 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526759 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 21:58:06.819908 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526762 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 21:58:06.820644 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526764 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 21:58:06.820644 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526767 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 21:58:06.820644 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526769 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 21:58:06.820644 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526772 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 21:58:06.820644 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526774 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 21:58:06.820644 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526777 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 21:58:06.820644 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526779 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 21:58:06.820644 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526782 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 21:58:06.820644 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526784 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 21:58:06.820644 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526787 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 21:58:06.820644 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526789 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 21:58:06.820644 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526793 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 21:58:06.820644 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526796 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 21:58:06.820644 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526800 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 21:58:06.820644 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526803 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 21:58:06.820644 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526805 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 21:58:06.820644 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526808 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 21:58:06.820644 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.526811 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 21:58:06.822814 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.527595 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 21:58:06.822814 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.535645 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 13 21:58:06.822814 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.535661 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 13 21:58:06.822814 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.535684 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 13 21:58:06.822814 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.535690 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 13 21:58:06.822814 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535745 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 21:58:06.822814 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535749 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 21:58:06.822814 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535753 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 21:58:06.822814 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535756 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 21:58:06.822814 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535759 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 21:58:06.822814 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535762 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 21:58:06.822814 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535764 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 21:58:06.822814 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535767 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 21:58:06.822814 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535771 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 21:58:06.823551 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535776 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 21:58:06.823551 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535780 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 21:58:06.823551 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535784 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 21:58:06.823551 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535787 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 21:58:06.823551 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535790 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 21:58:06.823551 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535792 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 21:58:06.823551 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535795 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 21:58:06.823551 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535797 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 21:58:06.823551 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535803 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 21:58:06.823551 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535807 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 21:58:06.823551 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535809 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 21:58:06.823551 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535812 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 21:58:06.823551 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535815 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 21:58:06.823551 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535818 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 21:58:06.823551 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535821 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 21:58:06.823551 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535823 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 21:58:06.823551 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535826 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 21:58:06.823551 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535829 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 21:58:06.823551 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535831 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 21:58:06.824298 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535834 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 21:58:06.824298 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535836 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 21:58:06.824298 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535839 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 21:58:06.824298 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535842 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 21:58:06.824298 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535845 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 21:58:06.824298 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535848 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 21:58:06.824298 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535850 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 21:58:06.824298 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535853 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 21:58:06.824298 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535855 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 21:58:06.824298 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535858 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 13 21:58:06.824298 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535861 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 21:58:06.824298 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535864 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 21:58:06.824298 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535866 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 21:58:06.824298 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535869 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 21:58:06.824298 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535873 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 21:58:06.824298 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535876 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 21:58:06.824298 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535879 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 21:58:06.824298 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535882 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 21:58:06.824298 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535885 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535887 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535890 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535894 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535896 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535899 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535901 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535904 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535906 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535909 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535911 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535914 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535916 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535919 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535921 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535924 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535927 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535930 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535932 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535935 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 21:58:06.825036 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535937 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 21:58:06.826525 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535940 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 21:58:06.826525 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535943 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 21:58:06.826525 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535945 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 21:58:06.826525 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535948 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 21:58:06.826525 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535950 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 21:58:06.826525 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535953 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 21:58:06.826525 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535955 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 21:58:06.826525 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535958 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 21:58:06.826525 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535960 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 21:58:06.826525 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535963 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 21:58:06.826525 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535966 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 21:58:06.826525 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535968 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 21:58:06.826525 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535971 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 21:58:06.826525 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535974 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 21:58:06.826525 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535978 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 21:58:06.826525 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535980 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 21:58:06.826525 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535983 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 21:58:06.826525 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535985 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 21:58:06.826525 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535990 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 21:58:06.827268 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535992 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 21:58:06.827268 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535995 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 21:58:06.827268 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.535997 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 21:58:06.827268 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536000 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 21:58:06.827268 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536002 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 21:58:06.827268 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536005 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 21:58:06.827268 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536008 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 21:58:06.827268 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536011 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 21:58:06.827268 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536014 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 21:58:06.827268 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536016 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 21:58:06.827268 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536019 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 21:58:06.827268 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536022 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 21:58:06.827268 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536024 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 21:58:06.827268 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.536029 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 21:58:06.827268 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536163 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536170 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536173 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536176 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536179 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536181 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536184 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536187 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536190 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536193 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536195 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536198 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536201 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536205 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536208 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536211 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536213 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536216 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536220 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536223 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 21:58:06.828724 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536226 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 21:58:06.829465 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536228 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 21:58:06.829465 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536231 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 21:58:06.829465 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536234 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 21:58:06.829465 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536236 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 21:58:06.829465 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536239 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 21:58:06.829465 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536241 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 21:58:06.829465 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536244 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 21:58:06.829465 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536248 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 21:58:06.829465 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536251 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 21:58:06.829465 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536254 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 21:58:06.829465 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536257 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 21:58:06.829465 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536260 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 21:58:06.829465 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536263 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 21:58:06.829465 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536265 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 21:58:06.829465 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536268 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 21:58:06.829465 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536271 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 21:58:06.829465 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536273 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 21:58:06.829465 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536276 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 21:58:06.829465 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536278 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 21:58:06.830936 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536281 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 21:58:06.830936 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536284 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 21:58:06.830936 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536288 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 21:58:06.830936 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536290 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 21:58:06.830936 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536293 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 21:58:06.830936 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536297 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 21:58:06.830936 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536300 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 21:58:06.830936 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536303 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 21:58:06.830936 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536305 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 21:58:06.830936 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536308 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 21:58:06.830936 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536312 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 21:58:06.830936 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536314 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 21:58:06.830936 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536317 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 21:58:06.830936 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536319 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 21:58:06.830936 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536322 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 21:58:06.830936 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536325 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 21:58:06.830936 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536327 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 21:58:06.830936 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536330 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 21:58:06.830936 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536333 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536335 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536338 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536340 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536343 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536346 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536348 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536351 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536354 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536356 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536359 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536362 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536364 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536367 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536369 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536372 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536374 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536376 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536379 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536383 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 21:58:06.831664 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536386 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536388 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536391 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536393 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536397 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536399 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536402 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536405 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536407 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536411 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536415 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536418 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536421 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536424 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536426 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536429 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536431 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536434 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536436 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536439 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 21:58:06.833163 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:06.536441 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 21:58:06.833903 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.536446 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 21:58:06.833903 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.536540 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jul 13 21:58:06.833903 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.538730 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 13 21:58:06.833903 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.540311 2569 server.go:1019] "Starting client certificate rotation" Jul 13 21:58:06.833903 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.540407 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 21:58:06.833903 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.540439 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 21:58:06.833903 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.543244 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 21:58:06.833903 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.547467 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 21:58:06.833903 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.559541 2569 log.go:25] "Validated CRI v1 runtime API" Jul 13 21:58:06.833903 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.562160 2569 log.go:25] "Validated CRI v1 image API" Jul 13 21:58:06.833903 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.563342 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 13 21:58:06.833903 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.569212 2569 fs.go:135] Filesystem UUIDs: map[3c76b041-d3ce-462c-966e-0dc8a5e42372:/dev/nvme0n1p4 7567111a-699c-4f10-8e4a-55751b857e02:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 13 21:58:06.834992 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.569233 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 13 21:58:06.835373 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.575421 2569 manager.go:216] Machine: {Timestamp:2026-07-13 21:58:06.573909998 +0000 UTC m=+0.382599887 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097519 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec261c8627d0abb93d27c6f2afa81de1 SystemUUID:ec261c86-27d0-abb9-3d27-c6f2afa81de1 BootID:9b55828b-5efa-41cc-b7d2-7fa367ec3b5f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:36:4a:33:b5:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:36:4a:33:b5:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:63:fa:0d:86:87 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 13 21:58:06.835373 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.575520 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 13 21:58:06.835373 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.575617 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 13 21:58:06.835373 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.579402 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 13 21:58:06.835373 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.579437 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-183.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 13 21:58:06.835373 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.580344 2569 topology_manager.go:138] "Creating topology manager with none policy" Jul 13 21:58:06.835373 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.580358 2569 container_manager_linux.go:309] "Creating device plugin manager" Jul 13 21:58:06.835373 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.580368 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.581109 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.582026 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.582898 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.586260 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.586384 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.586411 2569 kubelet.go:486] "Attempting to sync node with API server" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.586419 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.586432 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.586440 2569 kubelet.go:398] "Adding apiserver pod source" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.586448 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.586757 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.586778 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.591406 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.593614 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.593641 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.593803 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.593815 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.593821 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.593826 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.593835 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.593844 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.594432 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.594439 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.594446 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.594452 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.594470 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.594490 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.596181 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.596189 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:06.596458 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:06.596458 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.598936 2569 server.go:1268] "Started kubelet" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.599259 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.599946 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.599979 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.601459 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 13 21:58:06.837603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.610972 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-183.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:06.607002 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-183.ec2.internal.18c1f841432b8fb2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-183.ec2.internal,UID:ip-10-0-137-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-183.ec2.internal,},FirstTimestamp:2026-07-13 21:58:06.598918066 +0000 UTC m=+0.407607956,LastTimestamp:2026-07-13 21:58:06.598918066 +0000 UTC m=+0.407607956,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-183.ec2.internal,}" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.612316 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.612562 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:06.613094 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-183.ec2.internal\" not found" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.615129 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:06.615313 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.615392 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.615410 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.615776 2569 server.go:310] "Adding debug handlers to kubelet server" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.615803 2569 reconstruct.go:97] "Volume reconstruction finished" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.615815 2569 reconciler.go:29] "Reconciler: start to sync state" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.615775 2569 factory.go:55] Registering systemd factory Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.615872 2569 factory.go:223] Registration of the systemd container factory successfully Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.616132 2569 factory.go:153] Registering CRI-O factory Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.616142 2569 factory.go:223] Registration of the crio container factory successfully Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.616181 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.616200 2569 factory.go:103] Registering Raw factory Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.616214 2569 manager.go:1196] Started watching for new ooms in manager Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.616526 2569 manager.go:349] Starting recovery of all containers Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:06.622293 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:06.622338 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.624993 2569 manager.go:354] Recovery completed Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.631614 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.632433 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b6gbj" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.633390 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.633412 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.633431 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.633964 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.633963 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.634033 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.634068 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.634742 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:58:06.838711 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.634765 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.634780 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.635194 2569 policy_none.go:49] "None policy: Start" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.635215 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.635230 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.636248 2569 policy_none.go:47] "Start" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.640662 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b6gbj" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:06.643069 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-183.ec2.internal.18c1f8414539b6d7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-183.ec2.internal,UID:ip-10-0-137-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-183.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-183.ec2.internal,},FirstTimestamp:2026-07-13 21:58:06.633400023 +0000 UTC m=+0.442089912,LastTimestamp:2026-07-13 21:58:06.633400023 +0000 UTC m=+0.442089912,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-183.ec2.internal,}" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.682660 2569 manager.go:337] "Starting Device Plugin manager" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:06.682685 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.682694 2569 server.go:87] "Starting device plugin registration server" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.682873 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.682885 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.682956 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.683022 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.683030 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:06.683564 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:06.683611 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-183.ec2.internal\" not found" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.771549 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.772688 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.772731 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.772759 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:06.772798 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.775405 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.783382 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.784269 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.784295 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.784308 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.784331 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-183.ec2.internal" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:06.793167 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-183.ec2.internal" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:06.793185 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-183.ec2.internal\": node \"ip-10-0-137-183.ec2.internal\" not found" Jul 13 21:58:06.839803 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:06.820734 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-183.ec2.internal\" not found" Jul 13 21:58:06.930421 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:06.921082 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-183.ec2.internal\" not found" Jul 13 21:58:07.039420 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:07.021388 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-183.ec2.internal\" not found" Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:07.121778 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-183.ec2.internal\" not found" Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.126716 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-183.ec2.internal"] Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.126803 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.128516 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.128542 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.128552 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.130802 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.130931 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal" Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.130962 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.131562 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.131574 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.131585 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.131592 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.131595 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.131602 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.133632 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-183.ec2.internal" Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.133658 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.134319 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.134344 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:58:07.159655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.134358 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:58:07.259087 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:07.162720 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-183.ec2.internal\" not found" node="ip-10-0-137-183.ec2.internal" Jul 13 21:58:07.259087 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:07.166915 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-183.ec2.internal\" not found" node="ip-10-0-137-183.ec2.internal" Jul 13 21:58:07.259087 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.218943 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/161221439d14e33d352dfa94468aef89-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal\" (UID: \"161221439d14e33d352dfa94468aef89\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal" Jul 13 21:58:07.259087 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.218964 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/161221439d14e33d352dfa94468aef89-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal\" (UID: \"161221439d14e33d352dfa94468aef89\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal" Jul 13 21:58:07.259087 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.218981 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/645e95dea7e9a87e47a32d256b790eae-config\") pod \"kube-apiserver-proxy-ip-10-0-137-183.ec2.internal\" (UID: \"645e95dea7e9a87e47a32d256b790eae\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-183.ec2.internal" Jul 13 21:58:07.259087 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:07.222024 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-183.ec2.internal\" not found" Jul 13 21:58:07.372431 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.319155 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/161221439d14e33d352dfa94468aef89-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal\" (UID: \"161221439d14e33d352dfa94468aef89\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal" Jul 13 21:58:07.372431 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.319176 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/645e95dea7e9a87e47a32d256b790eae-config\") pod \"kube-apiserver-proxy-ip-10-0-137-183.ec2.internal\" (UID: \"645e95dea7e9a87e47a32d256b790eae\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-183.ec2.internal" Jul 13 21:58:07.372431 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.319192 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/161221439d14e33d352dfa94468aef89-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal\" (UID: \"161221439d14e33d352dfa94468aef89\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal" Jul 13 21:58:07.372431 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.319226 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/161221439d14e33d352dfa94468aef89-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal\" (UID: \"161221439d14e33d352dfa94468aef89\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal" Jul 13 21:58:07.372431 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.319258 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/645e95dea7e9a87e47a32d256b790eae-config\") pod \"kube-apiserver-proxy-ip-10-0-137-183.ec2.internal\" (UID: \"645e95dea7e9a87e47a32d256b790eae\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-183.ec2.internal" Jul 13 21:58:07.372431 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.319259 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/161221439d14e33d352dfa94468aef89-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal\" (UID: \"161221439d14e33d352dfa94468aef89\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal" Jul 13 21:58:07.372431 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:07.322268 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-183.ec2.internal\" not found" Jul 13 21:58:07.478841 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:07.422752 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-183.ec2.internal\" not found" Jul 13 21:58:07.478841 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.464890 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal" Jul 13 21:58:07.478841 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.469380 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-183.ec2.internal" Jul 13 21:58:08.729102 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:07.523237 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-183.ec2.internal\" not found" Jul 13 21:58:08.729102 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.524078 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 21:58:08.729102 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.540083 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 13 21:58:08.729102 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.540214 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 13 21:58:08.729102 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:07.540252 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 13 21:58:08.775418 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:08.775371 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-183.ec2.internal\" not found" Jul 13 21:58:08.795520 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:08.795479 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 21:53:07 +0000 UTC" deadline="2027-12-16 15:34:16.309951886 +0000 UTC" Jul 13 21:58:08.795520 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:08.795511 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12497h36m7.514443832s" Jul 13 21:58:08.798655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:08.798640 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:58:08.800125 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:08.800110 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:58:08.800195 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:08.800138 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:58:08.800195 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:08.800148 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:58:08.875455 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:08.875427 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-183.ec2.internal\" not found" Jul 13 21:58:08.975727 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:08.975707 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-183.ec2.internal\" not found" Jul 13 21:58:09.076281 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:09.076209 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-183.ec2.internal\" not found" Jul 13 21:58:09.107042 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.107025 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 21:58:09.176596 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:09.176567 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-183.ec2.internal\" not found" Jul 13 21:58:09.277384 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:09.277354 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-183.ec2.internal\" not found" Jul 13 21:58:09.337058 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.337014 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 21:58:09.366011 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.365989 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-183.ec2.internal" Jul 13 21:58:09.379355 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.379332 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 21:58:09.380201 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.380188 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal" Jul 13 21:58:09.383329 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:09.383300 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645e95dea7e9a87e47a32d256b790eae.slice/crio-f65e821a6316737600bf6cc9236571d8c036823521d5267b4a6d645e79d0555a WatchSource:0}: Error finding container f65e821a6316737600bf6cc9236571d8c036823521d5267b4a6d645e79d0555a: Status 404 returned error can't find the container with id f65e821a6316737600bf6cc9236571d8c036823521d5267b4a6d645e79d0555a Jul 13 21:58:09.383707 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:09.383689 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod161221439d14e33d352dfa94468aef89.slice/crio-5d12bf314252590bbc767c7f1617a0cf3046ecef4d9e92690d86eabc5b7cefe5 WatchSource:0}: Error finding container 5d12bf314252590bbc767c7f1617a0cf3046ecef4d9e92690d86eabc5b7cefe5: Status 404 returned error can't find the container with id 5d12bf314252590bbc767c7f1617a0cf3046ecef4d9e92690d86eabc5b7cefe5 Jul 13 21:58:09.388146 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.388130 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 21:58:09.394275 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.394258 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 21:58:09.739513 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.739441 2569 apiserver.go:52] "Watching apiserver" Jul 13 21:58:09.747865 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.747846 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 13 21:58:09.748177 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.748158 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-9jggp","openshift-cluster-node-tuning-operator/tuned-n9gc7","openshift-image-registry/node-ca-782jn","openshift-multus/multus-additional-cni-plugins-dwskc","openshift-multus/multus-rvqz8","openshift-multus/network-metrics-daemon-tj476","openshift-network-diagnostics/network-check-target-qmrjq","openshift-network-operator/iptables-alerter-g4fk2","kube-system/global-pull-secret-syncer-qv2xs","kube-system/konnectivity-agent-vpg2f","kube-system/kube-apiserver-proxy-ip-10-0-137-183.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2","openshift-dns/node-resolver-4m9ps","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal"] Jul 13 21:58:09.752041 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.752020 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.754120 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.754099 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.754726 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.754708 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 13 21:58:09.754820 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.754737 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 13 21:58:09.754872 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.754845 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-25qlw\"" Jul 13 21:58:09.754945 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.754930 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 13 21:58:09.756396 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.756377 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:09.756483 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:09.756451 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmrjq" podUID="2ee46b9a-7019-44e4-a021-5c4ff5182a65" Jul 13 21:58:09.756722 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.756702 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 13 21:58:09.756722 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.756717 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c5vbl\"" Jul 13 21:58:09.756857 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.756750 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 13 21:58:09.758607 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.758591 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.760905 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.760876 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.760905 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.760897 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 13 21:58:09.761045 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.760884 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ngg2p\"" Jul 13 21:58:09.761045 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.760990 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 13 21:58:09.761153 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.761063 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 13 21:58:09.761208 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.761159 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 13 21:58:09.761377 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.761357 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 13 21:58:09.763132 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.763115 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 13 21:58:09.763282 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.763267 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:09.763420 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:09.763396 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tj476" podUID="227fb8d9-c7c7-4722-bba7-f2c2fc9b482f" Jul 13 21:58:09.763509 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.763496 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8lhts\"" Jul 13 21:58:09.765768 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.765750 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.766860 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.766841 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 13 21:58:09.767941 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.767926 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g4fk2" Jul 13 21:58:09.768112 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.768096 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 13 21:58:09.768280 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.768258 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 13 21:58:09.768397 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.768314 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 13 21:58:09.768397 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.768346 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 13 21:58:09.768659 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.768645 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 13 21:58:09.768729 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.768681 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xvwcq\"" Jul 13 21:58:09.769465 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.769451 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 13 21:58:09.770270 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.770255 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:09.770356 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:09.770316 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv2xs" podUID="2b1adc79-d0de-44ab-b9c5-747c579b20cb" Jul 13 21:58:09.770416 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.770378 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 13 21:58:09.770416 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.770384 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k8kxb\"" Jul 13 21:58:09.770671 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.770638 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 13 21:58:09.770804 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.770788 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 13 21:58:09.772461 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.772442 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vpg2f" Jul 13 21:58:09.776221 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.776139 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-782jn" Jul 13 21:58:09.776650 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.776626 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-run-ovn\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.776789 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.776665 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-host\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.776789 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.776692 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-multus-cni-dir\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.776789 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.776714 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5820bad8-3692-4171-b5e1-9f165718fe67-socket-dir\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.776789 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.776742 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 13 21:58:09.776789 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.776771 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5820bad8-3692-4171-b5e1-9f165718fe67-device-dir\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.777036 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.776825 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qcrdl\"" Jul 13 21:58:09.777036 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.776825 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-sysctl-conf\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.777036 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.776907 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eab93415-1e38-4dbe-9c23-dbc2b0354a12-multus-daemon-config\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.777036 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.776928 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-etc-kubernetes\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.777036 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.776964 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83271b6e-2653-44d9-8ddb-03bed0f66e39-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.777036 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.776995 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5820bad8-3692-4171-b5e1-9f165718fe67-etc-selinux\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.777036 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777005 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 13 21:58:09.777036 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777016 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-lib-modules\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.777036 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777039 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-host-var-lib-kubelet\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.777416 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777074 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-cni-netd\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.777416 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777097 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-os-release\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.777416 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777130 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-host-run-k8s-cni-cncf-io\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.777416 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777154 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-host-var-lib-cni-bin\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.777416 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777184 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-host-var-lib-cni-multus\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.777416 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777238 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-hostroot\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.777416 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777268 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-cni-bin\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.777416 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777291 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-sysconfig\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.777416 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777313 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-tuned\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.777416 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777334 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-tmp\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.777416 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777359 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljtgj\" (UniqueName: \"kubernetes.io/projected/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-kube-api-access-ljtgj\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.777416 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777395 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlcms\" (UniqueName: \"kubernetes.io/projected/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-kube-api-access-rlcms\") pod \"network-metrics-daemon-tj476\" (UID: \"227fb8d9-c7c7-4722-bba7-f2c2fc9b482f\") " pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:09.777416 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777418 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-kubelet\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.777960 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777441 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-slash\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.777960 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777465 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-multus-socket-dir-parent\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.777960 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777487 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-run-systemd\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.777960 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777509 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b264r\" (UniqueName: \"kubernetes.io/projected/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-kube-api-access-b264r\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.777960 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777548 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5820bad8-3692-4171-b5e1-9f165718fe67-registration-dir\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.777960 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777576 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmvds\" (UniqueName: \"kubernetes.io/projected/5820bad8-3692-4171-b5e1-9f165718fe67-kube-api-access-wmvds\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.777960 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777605 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-multus-conf-dir\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.777960 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777653 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs\") pod \"network-metrics-daemon-tj476\" (UID: \"227fb8d9-c7c7-4722-bba7-f2c2fc9b482f\") " pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:09.777960 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777682 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-var-lib-openvswitch\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.777960 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777705 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-etc-openvswitch\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.777960 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777733 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-run-openvswitch\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.777960 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777757 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-systemd-units\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.777960 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777778 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-log-socket\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.777960 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777815 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-env-overrides\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.778440 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777845 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7340cc2c-3aa2-4811-916f-90f9e4b30ae3-iptables-alerter-script\") pod \"iptables-alerter-g4fk2\" (UID: \"7340cc2c-3aa2-4811-916f-90f9e4b30ae3\") " pod="openshift-network-operator/iptables-alerter-g4fk2" Jul 13 21:58:09.778440 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777867 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-sys\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.778440 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777895 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-node-log\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.778440 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777918 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-run-ovn-kubernetes\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.778440 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777941 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-ovnkube-config\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.778440 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777963 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg6sr\" (UniqueName: \"kubernetes.io/projected/7340cc2c-3aa2-4811-916f-90f9e4b30ae3-kube-api-access-vg6sr\") pod \"iptables-alerter-g4fk2\" (UID: \"7340cc2c-3aa2-4811-916f-90f9e4b30ae3\") " pod="openshift-network-operator/iptables-alerter-g4fk2" Jul 13 21:58:09.778440 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.777986 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-cnibin\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.778440 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778022 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pjb5\" (UniqueName: \"kubernetes.io/projected/eab93415-1e38-4dbe-9c23-dbc2b0354a12-kube-api-access-5pjb5\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.778440 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778047 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83271b6e-2653-44d9-8ddb-03bed0f66e39-system-cni-dir\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.778440 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778082 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/83271b6e-2653-44d9-8ddb-03bed0f66e39-os-release\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.778440 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778112 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-var-lib-kubelet\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.778440 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778135 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83271b6e-2653-44d9-8ddb-03bed0f66e39-cni-binary-copy\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.778440 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778154 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/83271b6e-2653-44d9-8ddb-03bed0f66e39-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.779031 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778169 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfw2t\" (UniqueName: \"kubernetes.io/projected/83271b6e-2653-44d9-8ddb-03bed0f66e39-kube-api-access-hfw2t\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.779031 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778191 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5820bad8-3692-4171-b5e1-9f165718fe67-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.779031 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778213 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eab93415-1e38-4dbe-9c23-dbc2b0354a12-cni-binary-copy\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.779031 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778231 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7340cc2c-3aa2-4811-916f-90f9e4b30ae3-host-slash\") pod \"iptables-alerter-g4fk2\" (UID: \"7340cc2c-3aa2-4811-916f-90f9e4b30ae3\") " pod="openshift-network-operator/iptables-alerter-g4fk2" Jul 13 21:58:09.779031 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778243 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-run-netns\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.779031 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778274 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5820bad8-3692-4171-b5e1-9f165718fe67-sys-fs\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.779031 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778291 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-modprobe-d\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.779031 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778320 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-kubernetes\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.779031 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778349 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-sysctl-d\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.779031 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778365 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-host-run-netns\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.779031 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778379 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83271b6e-2653-44d9-8ddb-03bed0f66e39-cnibin\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.779031 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778408 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-systemd\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.779031 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778442 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp64d\" (UniqueName: \"kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d\") pod \"network-check-target-qmrjq\" (UID: \"2ee46b9a-7019-44e4-a021-5c4ff5182a65\") " pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:09.779693 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778470 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.779693 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778497 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-run\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.779693 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778518 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-host-run-multus-certs\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.779693 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778556 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-ovn-node-metrics-cert\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.779693 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778585 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-ovnkube-script-lib\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.779693 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778606 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-system-cni-dir\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.779693 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778629 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/83271b6e-2653-44d9-8ddb-03bed0f66e39-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.779693 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778783 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 13 21:58:09.779693 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.778903 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4m9ps" Jul 13 21:58:09.779693 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.779078 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 13 21:58:09.779693 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.779121 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 13 21:58:09.779693 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.779347 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bd99z\"" Jul 13 21:58:09.781607 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.781437 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 13 21:58:09.781607 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.781437 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 13 21:58:09.781607 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.781506 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nldqr\"" Jul 13 21:58:09.781823 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.781806 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 21:58:09.798610 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.798589 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-48lc8" Jul 13 21:58:09.804407 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.804387 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 21:58:09.809932 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.809912 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-48lc8" Jul 13 21:58:09.867643 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.867575 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 13 21:58:09.879481 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.879453 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlcms\" (UniqueName: \"kubernetes.io/projected/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-kube-api-access-rlcms\") pod \"network-metrics-daemon-tj476\" (UID: \"227fb8d9-c7c7-4722-bba7-f2c2fc9b482f\") " pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:09.879611 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.879492 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-kubelet\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.879611 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.879520 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-slash\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.879611 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.879565 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-multus-socket-dir-parent\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.879611 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.879588 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-run-systemd\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.879834 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.879617 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b264r\" (UniqueName: \"kubernetes.io/projected/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-kube-api-access-b264r\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.879834 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.879644 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b1adc79-d0de-44ab-b9c5-747c579b20cb-dbus\") pod \"global-pull-secret-syncer-qv2xs\" (UID: \"2b1adc79-d0de-44ab-b9c5-747c579b20cb\") " pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:09.879834 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.879703 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f33a38a4-ac69-4c8f-915f-2e84fa701683-tmp-dir\") pod \"node-resolver-4m9ps\" (UID: \"f33a38a4-ac69-4c8f-915f-2e84fa701683\") " pod="openshift-dns/node-resolver-4m9ps" Jul 13 21:58:09.879834 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.879736 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5820bad8-3692-4171-b5e1-9f165718fe67-registration-dir\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.879834 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.879762 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmvds\" (UniqueName: \"kubernetes.io/projected/5820bad8-3692-4171-b5e1-9f165718fe67-kube-api-access-wmvds\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.879834 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.879782 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-multus-conf-dir\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.879834 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.879797 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs\") pod \"network-metrics-daemon-tj476\" (UID: \"227fb8d9-c7c7-4722-bba7-f2c2fc9b482f\") " pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:09.879834 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.879825 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-var-lib-openvswitch\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.880202 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.879844 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-etc-openvswitch\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.880202 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.879866 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-run-openvswitch\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.880202 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.879892 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-systemd-units\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.880202 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.879913 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-log-socket\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.880202 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.879938 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-env-overrides\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.880202 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.879963 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7340cc2c-3aa2-4811-916f-90f9e4b30ae3-iptables-alerter-script\") pod \"iptables-alerter-g4fk2\" (UID: \"7340cc2c-3aa2-4811-916f-90f9e4b30ae3\") " pod="openshift-network-operator/iptables-alerter-g4fk2" Jul 13 21:58:09.880202 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880182 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-sys\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.881640 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880214 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-node-log\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.881640 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880241 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-run-ovn-kubernetes\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.881640 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880266 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-ovnkube-config\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.881640 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880291 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg6sr\" (UniqueName: \"kubernetes.io/projected/7340cc2c-3aa2-4811-916f-90f9e4b30ae3-kube-api-access-vg6sr\") pod \"iptables-alerter-g4fk2\" (UID: \"7340cc2c-3aa2-4811-916f-90f9e4b30ae3\") " pod="openshift-network-operator/iptables-alerter-g4fk2" Jul 13 21:58:09.881640 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880315 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-cnibin\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.881640 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880341 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pjb5\" (UniqueName: \"kubernetes.io/projected/eab93415-1e38-4dbe-9c23-dbc2b0354a12-kube-api-access-5pjb5\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.881640 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880347 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-multus-conf-dir\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.881640 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880365 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83271b6e-2653-44d9-8ddb-03bed0f66e39-system-cni-dir\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.881640 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880391 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/83271b6e-2653-44d9-8ddb-03bed0f66e39-os-release\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.881640 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880425 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f33a38a4-ac69-4c8f-915f-2e84fa701683-hosts-file\") pod \"node-resolver-4m9ps\" (UID: \"f33a38a4-ac69-4c8f-915f-2e84fa701683\") " pod="openshift-dns/node-resolver-4m9ps" Jul 13 21:58:09.881640 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880433 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-kubelet\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.881640 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880462 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-var-lib-kubelet\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.881640 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:09.880494 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:58:09.881640 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880519 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83271b6e-2653-44d9-8ddb-03bed0f66e39-cni-binary-copy\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.881640 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880521 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-var-lib-kubelet\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.882421 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880575 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-sys\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.882421 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:09.880573 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs podName:227fb8d9-c7c7-4722-bba7-f2c2fc9b482f nodeName:}" failed. No retries permitted until 2026-07-13 21:58:10.380552626 +0000 UTC m=+3.037757372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs") pod "network-metrics-daemon-tj476" (UID: "227fb8d9-c7c7-4722-bba7-f2c2fc9b482f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:58:09.882421 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880603 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-slash\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.882421 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880613 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/83271b6e-2653-44d9-8ddb-03bed0f66e39-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.882421 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880643 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0-konnectivity-ca\") pod \"konnectivity-agent-vpg2f\" (UID: \"3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0\") " pod="kube-system/konnectivity-agent-vpg2f" Jul 13 21:58:09.882421 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880660 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-multus-socket-dir-parent\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.882421 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880670 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfw2t\" (UniqueName: \"kubernetes.io/projected/83271b6e-2653-44d9-8ddb-03bed0f66e39-kube-api-access-hfw2t\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.882421 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880706 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-run-systemd\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.882421 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880737 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret\") pod \"global-pull-secret-syncer-qv2xs\" (UID: \"2b1adc79-d0de-44ab-b9c5-747c579b20cb\") " pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:09.882421 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880763 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-var-lib-openvswitch\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.882421 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880852 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5820bad8-3692-4171-b5e1-9f165718fe67-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.882421 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880900 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-node-log\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.882421 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881080 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5820bad8-3692-4171-b5e1-9f165718fe67-registration-dir\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.883083 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881207 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7340cc2c-3aa2-4811-916f-90f9e4b30ae3-iptables-alerter-script\") pod \"iptables-alerter-g4fk2\" (UID: \"7340cc2c-3aa2-4811-916f-90f9e4b30ae3\") " pod="openshift-network-operator/iptables-alerter-g4fk2" Jul 13 21:58:09.883083 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881305 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83271b6e-2653-44d9-8ddb-03bed0f66e39-system-cni-dir\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.883083 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881389 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-log-socket\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.883083 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881437 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-etc-openvswitch\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.883083 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881481 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-run-openvswitch\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.883083 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881522 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-systemd-units\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.883083 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881604 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-ovnkube-config\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.883083 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880766 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5820bad8-3692-4171-b5e1-9f165718fe67-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.883083 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881672 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83271b6e-2653-44d9-8ddb-03bed0f66e39-cni-binary-copy\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.883083 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881713 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eab93415-1e38-4dbe-9c23-dbc2b0354a12-cni-binary-copy\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.883083 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881735 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-cnibin\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.883083 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881762 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7340cc2c-3aa2-4811-916f-90f9e4b30ae3-host-slash\") pod \"iptables-alerter-g4fk2\" (UID: \"7340cc2c-3aa2-4811-916f-90f9e4b30ae3\") " pod="openshift-network-operator/iptables-alerter-g4fk2" Jul 13 21:58:09.883083 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881760 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/83271b6e-2653-44d9-8ddb-03bed0f66e39-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.883083 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881794 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-run-netns\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.883794 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881824 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/83271b6e-2653-44d9-8ddb-03bed0f66e39-os-release\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.883794 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881819 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5820bad8-3692-4171-b5e1-9f165718fe67-sys-fs\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.883794 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.880900 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-run-ovn-kubernetes\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.883794 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881864 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-modprobe-d\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.883794 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881878 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5820bad8-3692-4171-b5e1-9f165718fe67-sys-fs\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.883794 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881883 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7340cc2c-3aa2-4811-916f-90f9e4b30ae3-host-slash\") pod \"iptables-alerter-g4fk2\" (UID: \"7340cc2c-3aa2-4811-916f-90f9e4b30ae3\") " pod="openshift-network-operator/iptables-alerter-g4fk2" Jul 13 21:58:09.883794 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881889 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-kubernetes\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.883794 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881933 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-kubernetes\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.883794 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881935 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-env-overrides\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.883794 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881925 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-run-netns\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.883794 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.881978 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-sysctl-d\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.883794 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882007 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-host-run-netns\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.883794 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882059 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-host-run-netns\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.883794 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882074 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-modprobe-d\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.883794 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882104 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83271b6e-2653-44d9-8ddb-03bed0f66e39-cnibin\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.884530 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882125 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-sysctl-d\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.884530 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882152 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-systemd\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.884530 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882166 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83271b6e-2653-44d9-8ddb-03bed0f66e39-cnibin\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.884530 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882178 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp64d\" (UniqueName: \"kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d\") pod \"network-check-target-qmrjq\" (UID: \"2ee46b9a-7019-44e4-a021-5c4ff5182a65\") " pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:09.884530 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882203 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.884530 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882233 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a16c43cb-24ec-420c-953f-204d3584929c-host\") pod \"node-ca-782jn\" (UID: \"a16c43cb-24ec-420c-953f-204d3584929c\") " pod="openshift-image-registry/node-ca-782jn" Jul 13 21:58:09.884530 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882257 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.884530 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882260 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8qwj\" (UniqueName: \"kubernetes.io/projected/a16c43cb-24ec-420c-953f-204d3584929c-kube-api-access-j8qwj\") pod \"node-ca-782jn\" (UID: \"a16c43cb-24ec-420c-953f-204d3584929c\") " pod="openshift-image-registry/node-ca-782jn" Jul 13 21:58:09.884530 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882207 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-systemd\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.884530 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882313 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-run\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.884530 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882353 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-host-run-multus-certs\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.884530 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882376 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-ovn-node-metrics-cert\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.884530 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882377 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eab93415-1e38-4dbe-9c23-dbc2b0354a12-cni-binary-copy\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.884530 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882400 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-ovnkube-script-lib\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.885197 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882425 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-system-cni-dir\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.885197 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882449 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/83271b6e-2653-44d9-8ddb-03bed0f66e39-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.885197 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882477 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-run\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.885197 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882526 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-run-ovn\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.885197 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882586 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-host-run-multus-certs\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.885197 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882480 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-run-ovn\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.885197 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882623 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-host\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.885197 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882675 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-multus-cni-dir\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.885197 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882718 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5820bad8-3692-4171-b5e1-9f165718fe67-socket-dir\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.885197 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882771 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5820bad8-3692-4171-b5e1-9f165718fe67-device-dir\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.885197 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882796 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-sysctl-conf\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.885197 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882817 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 13 21:58:09.885197 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882828 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eab93415-1e38-4dbe-9c23-dbc2b0354a12-multus-daemon-config\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.885197 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882900 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-etc-kubernetes\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.885197 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882922 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83271b6e-2653-44d9-8ddb-03bed0f66e39-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.885910 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882945 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5820bad8-3692-4171-b5e1-9f165718fe67-etc-selinux\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.885910 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882969 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-lib-modules\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.885910 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.882995 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-host-var-lib-kubelet\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.885910 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883002 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-ovnkube-script-lib\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.885910 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883018 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-cni-netd\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.885910 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883045 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b1adc79-d0de-44ab-b9c5-747c579b20cb-kubelet-config\") pod \"global-pull-secret-syncer-qv2xs\" (UID: \"2b1adc79-d0de-44ab-b9c5-747c579b20cb\") " pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:09.885910 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883074 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-system-cni-dir\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.885910 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883082 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a16c43cb-24ec-420c-953f-204d3584929c-serviceca\") pod \"node-ca-782jn\" (UID: \"a16c43cb-24ec-420c-953f-204d3584929c\") " pod="openshift-image-registry/node-ca-782jn" Jul 13 21:58:09.885910 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883118 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-os-release\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.885910 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883209 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-host-run-k8s-cni-cncf-io\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.885910 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883234 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-host-var-lib-cni-bin\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.885910 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883280 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-host-var-lib-cni-multus\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.885910 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883336 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83271b6e-2653-44d9-8ddb-03bed0f66e39-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.885910 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883357 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-host-var-lib-cni-multus\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.885910 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883381 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-hostroot\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.886708 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883406 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-cni-bin\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.886708 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883415 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5820bad8-3692-4171-b5e1-9f165718fe67-etc-selinux\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.886708 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883464 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwlf8\" (UniqueName: \"kubernetes.io/projected/f33a38a4-ac69-4c8f-915f-2e84fa701683-kube-api-access-qwlf8\") pod \"node-resolver-4m9ps\" (UID: \"f33a38a4-ac69-4c8f-915f-2e84fa701683\") " pod="openshift-dns/node-resolver-4m9ps" Jul 13 21:58:09.886708 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883493 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-sysconfig\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.886708 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883546 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-tuned\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.886708 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883503 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-multus-cni-dir\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.886708 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883569 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-tmp\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.886708 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883588 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/83271b6e-2653-44d9-8ddb-03bed0f66e39-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.886708 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883644 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-host\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.886708 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883678 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5820bad8-3692-4171-b5e1-9f165718fe67-socket-dir\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.886708 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883720 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-os-release\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.886708 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883728 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljtgj\" (UniqueName: \"kubernetes.io/projected/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-kube-api-access-ljtgj\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.886708 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883756 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0-agent-certs\") pod \"konnectivity-agent-vpg2f\" (UID: \"3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0\") " pod="kube-system/konnectivity-agent-vpg2f" Jul 13 21:58:09.886708 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883761 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-host-run-k8s-cni-cncf-io\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.886708 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883817 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-host-var-lib-cni-bin\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.887283 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883836 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5820bad8-3692-4171-b5e1-9f165718fe67-device-dir\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.887283 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883879 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-etc-kubernetes\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.887283 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.883564 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-lib-modules\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.887283 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.884054 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-host-var-lib-kubelet\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.887283 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.884108 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-cni-netd\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.887283 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.884215 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-sysctl-conf\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.887283 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.884258 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-host-cni-bin\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.887283 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.884256 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eab93415-1e38-4dbe-9c23-dbc2b0354a12-hostroot\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.887283 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.884360 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-sysconfig\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.887283 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.884870 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eab93415-1e38-4dbe-9c23-dbc2b0354a12-multus-daemon-config\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.887697 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.887616 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-tmp\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.887792 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.887768 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-etc-tuned\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.889825 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.889736 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-ovn-node-metrics-cert\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.892490 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.891717 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlcms\" (UniqueName: \"kubernetes.io/projected/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-kube-api-access-rlcms\") pod \"network-metrics-daemon-tj476\" (UID: \"227fb8d9-c7c7-4722-bba7-f2c2fc9b482f\") " pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:09.892490 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:09.891754 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 21:58:09.892490 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:09.891774 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 21:58:09.892490 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:09.891787 2569 projected.go:196] Error preparing data for projected volume kube-api-access-xp64d for pod openshift-network-diagnostics/network-check-target-qmrjq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:58:09.892490 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:09.891856 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d podName:2ee46b9a-7019-44e4-a021-5c4ff5182a65 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:10.391836635 +0000 UTC m=+3.049041383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xp64d" (UniqueName: "kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d") pod "network-check-target-qmrjq" (UID: "2ee46b9a-7019-44e4-a021-5c4ff5182a65") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:58:09.892490 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.892221 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pjb5\" (UniqueName: \"kubernetes.io/projected/eab93415-1e38-4dbe-9c23-dbc2b0354a12-kube-api-access-5pjb5\") pod \"multus-rvqz8\" (UID: \"eab93415-1e38-4dbe-9c23-dbc2b0354a12\") " pod="openshift-multus/multus-rvqz8" Jul 13 21:58:09.895739 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.895696 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg6sr\" (UniqueName: \"kubernetes.io/projected/7340cc2c-3aa2-4811-916f-90f9e4b30ae3-kube-api-access-vg6sr\") pod \"iptables-alerter-g4fk2\" (UID: \"7340cc2c-3aa2-4811-916f-90f9e4b30ae3\") " pod="openshift-network-operator/iptables-alerter-g4fk2" Jul 13 21:58:09.895831 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.895660 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b264r\" (UniqueName: \"kubernetes.io/projected/d11fbab3-afd4-4eaa-9b10-c0262d17fc6d-kube-api-access-b264r\") pod \"ovnkube-node-9jggp\" (UID: \"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:09.896656 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.896596 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfw2t\" (UniqueName: \"kubernetes.io/projected/83271b6e-2653-44d9-8ddb-03bed0f66e39-kube-api-access-hfw2t\") pod \"multus-additional-cni-plugins-dwskc\" (UID: \"83271b6e-2653-44d9-8ddb-03bed0f66e39\") " pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:09.897956 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.897913 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmvds\" (UniqueName: \"kubernetes.io/projected/5820bad8-3692-4171-b5e1-9f165718fe67-kube-api-access-wmvds\") pod \"aws-ebs-csi-driver-node-hcrs2\" (UID: \"5820bad8-3692-4171-b5e1-9f165718fe67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:09.898420 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.898386 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljtgj\" (UniqueName: \"kubernetes.io/projected/a9b8b1b9-b95a-47bc-8f91-23012584c3e3-kube-api-access-ljtgj\") pod \"tuned-n9gc7\" (UID: \"a9b8b1b9-b95a-47bc-8f91-23012584c3e3\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:09.929867 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.929825 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal" event={"ID":"161221439d14e33d352dfa94468aef89","Type":"ContainerStarted","Data":"5d12bf314252590bbc767c7f1617a0cf3046ecef4d9e92690d86eabc5b7cefe5"} Jul 13 21:58:09.931231 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.931205 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-183.ec2.internal" event={"ID":"645e95dea7e9a87e47a32d256b790eae","Type":"ContainerStarted","Data":"f65e821a6316737600bf6cc9236571d8c036823521d5267b4a6d645e79d0555a"} Jul 13 21:58:09.986526 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.984160 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f33a38a4-ac69-4c8f-915f-2e84fa701683-hosts-file\") pod \"node-resolver-4m9ps\" (UID: \"f33a38a4-ac69-4c8f-915f-2e84fa701683\") " pod="openshift-dns/node-resolver-4m9ps" Jul 13 21:58:09.986526 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.984200 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0-konnectivity-ca\") pod \"konnectivity-agent-vpg2f\" (UID: \"3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0\") " pod="kube-system/konnectivity-agent-vpg2f" Jul 13 21:58:09.986526 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.984229 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret\") pod \"global-pull-secret-syncer-qv2xs\" (UID: \"2b1adc79-d0de-44ab-b9c5-747c579b20cb\") " pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:09.986526 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.984274 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a16c43cb-24ec-420c-953f-204d3584929c-host\") pod \"node-ca-782jn\" (UID: \"a16c43cb-24ec-420c-953f-204d3584929c\") " pod="openshift-image-registry/node-ca-782jn" Jul 13 21:58:09.986526 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.984297 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8qwj\" (UniqueName: \"kubernetes.io/projected/a16c43cb-24ec-420c-953f-204d3584929c-kube-api-access-j8qwj\") pod \"node-ca-782jn\" (UID: \"a16c43cb-24ec-420c-953f-204d3584929c\") " pod="openshift-image-registry/node-ca-782jn" Jul 13 21:58:09.986526 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.984333 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b1adc79-d0de-44ab-b9c5-747c579b20cb-kubelet-config\") pod \"global-pull-secret-syncer-qv2xs\" (UID: \"2b1adc79-d0de-44ab-b9c5-747c579b20cb\") " pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:09.986526 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.984354 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a16c43cb-24ec-420c-953f-204d3584929c-serviceca\") pod \"node-ca-782jn\" (UID: \"a16c43cb-24ec-420c-953f-204d3584929c\") " pod="openshift-image-registry/node-ca-782jn" Jul 13 21:58:09.986526 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.984402 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwlf8\" (UniqueName: \"kubernetes.io/projected/f33a38a4-ac69-4c8f-915f-2e84fa701683-kube-api-access-qwlf8\") pod \"node-resolver-4m9ps\" (UID: \"f33a38a4-ac69-4c8f-915f-2e84fa701683\") " pod="openshift-dns/node-resolver-4m9ps" Jul 13 21:58:09.986526 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.984428 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0-agent-certs\") pod \"konnectivity-agent-vpg2f\" (UID: \"3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0\") " pod="kube-system/konnectivity-agent-vpg2f" Jul 13 21:58:09.986526 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.984457 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b1adc79-d0de-44ab-b9c5-747c579b20cb-dbus\") pod \"global-pull-secret-syncer-qv2xs\" (UID: \"2b1adc79-d0de-44ab-b9c5-747c579b20cb\") " pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:09.986526 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.984484 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f33a38a4-ac69-4c8f-915f-2e84fa701683-tmp-dir\") pod \"node-resolver-4m9ps\" (UID: \"f33a38a4-ac69-4c8f-915f-2e84fa701683\") " pod="openshift-dns/node-resolver-4m9ps" Jul 13 21:58:09.986526 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.984838 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f33a38a4-ac69-4c8f-915f-2e84fa701683-tmp-dir\") pod \"node-resolver-4m9ps\" (UID: \"f33a38a4-ac69-4c8f-915f-2e84fa701683\") " pod="openshift-dns/node-resolver-4m9ps" Jul 13 21:58:09.986526 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.984905 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f33a38a4-ac69-4c8f-915f-2e84fa701683-hosts-file\") pod \"node-resolver-4m9ps\" (UID: \"f33a38a4-ac69-4c8f-915f-2e84fa701683\") " pod="openshift-dns/node-resolver-4m9ps" Jul 13 21:58:09.986526 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.985375 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0-konnectivity-ca\") pod \"konnectivity-agent-vpg2f\" (UID: \"3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0\") " pod="kube-system/konnectivity-agent-vpg2f" Jul 13 21:58:09.986526 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:09.985463 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 21:58:09.987281 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:09.985513 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret podName:2b1adc79-d0de-44ab-b9c5-747c579b20cb nodeName:}" failed. No retries permitted until 2026-07-13 21:58:10.485496325 +0000 UTC m=+3.142701088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret") pod "global-pull-secret-syncer-qv2xs" (UID: "2b1adc79-d0de-44ab-b9c5-747c579b20cb") : object "kube-system"/"original-pull-secret" not registered Jul 13 21:58:09.987281 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.985914 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a16c43cb-24ec-420c-953f-204d3584929c-serviceca\") pod \"node-ca-782jn\" (UID: \"a16c43cb-24ec-420c-953f-204d3584929c\") " pod="openshift-image-registry/node-ca-782jn" Jul 13 21:58:09.987281 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.985971 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a16c43cb-24ec-420c-953f-204d3584929c-host\") pod \"node-ca-782jn\" (UID: \"a16c43cb-24ec-420c-953f-204d3584929c\") " pod="openshift-image-registry/node-ca-782jn" Jul 13 21:58:09.987281 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.986205 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b1adc79-d0de-44ab-b9c5-747c579b20cb-kubelet-config\") pod \"global-pull-secret-syncer-qv2xs\" (UID: \"2b1adc79-d0de-44ab-b9c5-747c579b20cb\") " pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:09.987281 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.986798 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b1adc79-d0de-44ab-b9c5-747c579b20cb-dbus\") pod \"global-pull-secret-syncer-qv2xs\" (UID: \"2b1adc79-d0de-44ab-b9c5-747c579b20cb\") " pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:09.990025 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.989934 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0-agent-certs\") pod \"konnectivity-agent-vpg2f\" (UID: \"3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0\") " pod="kube-system/konnectivity-agent-vpg2f" Jul 13 21:58:09.994625 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.994448 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwlf8\" (UniqueName: \"kubernetes.io/projected/f33a38a4-ac69-4c8f-915f-2e84fa701683-kube-api-access-qwlf8\") pod \"node-resolver-4m9ps\" (UID: \"f33a38a4-ac69-4c8f-915f-2e84fa701683\") " pod="openshift-dns/node-resolver-4m9ps" Jul 13 21:58:09.996803 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:09.996781 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8qwj\" (UniqueName: \"kubernetes.io/projected/a16c43cb-24ec-420c-953f-204d3584929c-kube-api-access-j8qwj\") pod \"node-ca-782jn\" (UID: \"a16c43cb-24ec-420c-953f-204d3584929c\") " pod="openshift-image-registry/node-ca-782jn" Jul 13 21:58:10.061776 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.061623 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" Jul 13 21:58:10.067663 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.067641 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" Jul 13 21:58:10.076372 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.076319 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dwskc" Jul 13 21:58:10.078412 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:10.078384 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5820bad8_3692_4171_b5e1_9f165718fe67.slice/crio-7a8f63ccac0b417b729342e27200674d1265c2fcb5af5b294d40778d6109d7e4 WatchSource:0}: Error finding container 7a8f63ccac0b417b729342e27200674d1265c2fcb5af5b294d40778d6109d7e4: Status 404 returned error can't find the container with id 7a8f63ccac0b417b729342e27200674d1265c2fcb5af5b294d40778d6109d7e4 Jul 13 21:58:10.081265 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:10.081242 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b8b1b9_b95a_47bc_8f91_23012584c3e3.slice/crio-1d10968f854232d44154d7c4c45f7c10ea1006881c91bf82c71bf505684b185b WatchSource:0}: Error finding container 1d10968f854232d44154d7c4c45f7c10ea1006881c91bf82c71bf505684b185b: Status 404 returned error can't find the container with id 1d10968f854232d44154d7c4c45f7c10ea1006881c91bf82c71bf505684b185b Jul 13 21:58:10.083005 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.082839 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rvqz8" Jul 13 21:58:10.088826 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:10.088806 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83271b6e_2653_44d9_8ddb_03bed0f66e39.slice/crio-81abbd329cf219db8cf54b462946e86d10c71ace2fdeee81cb8785af287ec6b1 WatchSource:0}: Error finding container 81abbd329cf219db8cf54b462946e86d10c71ace2fdeee81cb8785af287ec6b1: Status 404 returned error can't find the container with id 81abbd329cf219db8cf54b462946e86d10c71ace2fdeee81cb8785af287ec6b1 Jul 13 21:58:10.089988 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.089969 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:10.092848 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:10.092830 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab93415_1e38_4dbe_9c23_dbc2b0354a12.slice/crio-d5cca99080fc2bd54c77b79951919120d9cebe630e6d91c6497f16cd910f3a4d WatchSource:0}: Error finding container d5cca99080fc2bd54c77b79951919120d9cebe630e6d91c6497f16cd910f3a4d: Status 404 returned error can't find the container with id d5cca99080fc2bd54c77b79951919120d9cebe630e6d91c6497f16cd910f3a4d Jul 13 21:58:10.098649 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.098284 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g4fk2" Jul 13 21:58:10.100849 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:10.100826 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11fbab3_afd4_4eaa_9b10_c0262d17fc6d.slice/crio-bfc858197a1e10c17164204d01d9fb9e0974632e7ca87d781e7ce5b65395f93e WatchSource:0}: Error finding container bfc858197a1e10c17164204d01d9fb9e0974632e7ca87d781e7ce5b65395f93e: Status 404 returned error can't find the container with id bfc858197a1e10c17164204d01d9fb9e0974632e7ca87d781e7ce5b65395f93e Jul 13 21:58:10.107075 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.107054 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vpg2f" Jul 13 21:58:10.109485 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:10.109421 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7340cc2c_3aa2_4811_916f_90f9e4b30ae3.slice/crio-431391c4a0fd3cba6e2d03f70560f80c0cb77b6111139fd2e9504feb72c640c2 WatchSource:0}: Error finding container 431391c4a0fd3cba6e2d03f70560f80c0cb77b6111139fd2e9504feb72c640c2: Status 404 returned error can't find the container with id 431391c4a0fd3cba6e2d03f70560f80c0cb77b6111139fd2e9504feb72c640c2 Jul 13 21:58:10.116481 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.116332 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-782jn" Jul 13 21:58:10.120650 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.120635 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4m9ps" Jul 13 21:58:10.134975 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:10.134950 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33a38a4_ac69_4c8f_915f_2e84fa701683.slice/crio-fbef3a4597f163cb576a0835174e49e65c9187c490746eac380987f1720fd6b7 WatchSource:0}: Error finding container fbef3a4597f163cb576a0835174e49e65c9187c490746eac380987f1720fd6b7: Status 404 returned error can't find the container with id fbef3a4597f163cb576a0835174e49e65c9187c490746eac380987f1720fd6b7 Jul 13 21:58:10.387615 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.387530 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs\") pod \"network-metrics-daemon-tj476\" (UID: \"227fb8d9-c7c7-4722-bba7-f2c2fc9b482f\") " pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:10.387758 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:10.387656 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:58:10.387758 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:10.387721 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs podName:227fb8d9-c7c7-4722-bba7-f2c2fc9b482f nodeName:}" failed. No retries permitted until 2026-07-13 21:58:11.387705841 +0000 UTC m=+4.044910604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs") pod "network-metrics-daemon-tj476" (UID: "227fb8d9-c7c7-4722-bba7-f2c2fc9b482f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:58:10.488774 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.488731 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret\") pod \"global-pull-secret-syncer-qv2xs\" (UID: \"2b1adc79-d0de-44ab-b9c5-747c579b20cb\") " pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:10.488997 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.488806 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp64d\" (UniqueName: \"kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d\") pod \"network-check-target-qmrjq\" (UID: \"2ee46b9a-7019-44e4-a021-5c4ff5182a65\") " pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:10.488997 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:10.488891 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 21:58:10.488997 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:10.488958 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 21:58:10.488997 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:10.488971 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret podName:2b1adc79-d0de-44ab-b9c5-747c579b20cb nodeName:}" failed. No retries permitted until 2026-07-13 21:58:11.488952068 +0000 UTC m=+4.146156831 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret") pod "global-pull-secret-syncer-qv2xs" (UID: "2b1adc79-d0de-44ab-b9c5-747c579b20cb") : object "kube-system"/"original-pull-secret" not registered Jul 13 21:58:10.488997 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:10.488977 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 21:58:10.489253 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:10.489005 2569 projected.go:196] Error preparing data for projected volume kube-api-access-xp64d for pod openshift-network-diagnostics/network-check-target-qmrjq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:58:10.489253 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:10.489048 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d podName:2ee46b9a-7019-44e4-a021-5c4ff5182a65 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:11.489037763 +0000 UTC m=+4.146242509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xp64d" (UniqueName: "kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d") pod "network-check-target-qmrjq" (UID: "2ee46b9a-7019-44e4-a021-5c4ff5182a65") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:58:10.812916 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.812841 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 21:53:09 +0000 UTC" deadline="2027-12-09 13:05:35.344764061 +0000 UTC" Jul 13 21:58:10.812916 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.812867 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12327h7m24.531900255s" Jul 13 21:58:10.933772 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.933740 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vpg2f" event={"ID":"3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0","Type":"ContainerStarted","Data":"c5d0d1851260c91c2593c34a699d88d9a88e009454927fc07192c91a90a55e83"} Jul 13 21:58:10.934894 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.934867 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g4fk2" event={"ID":"7340cc2c-3aa2-4811-916f-90f9e4b30ae3","Type":"ContainerStarted","Data":"431391c4a0fd3cba6e2d03f70560f80c0cb77b6111139fd2e9504feb72c640c2"} Jul 13 21:58:10.936032 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.936006 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" event={"ID":"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d","Type":"ContainerStarted","Data":"bfc858197a1e10c17164204d01d9fb9e0974632e7ca87d781e7ce5b65395f93e"} Jul 13 21:58:10.937150 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.937125 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rvqz8" event={"ID":"eab93415-1e38-4dbe-9c23-dbc2b0354a12","Type":"ContainerStarted","Data":"d5cca99080fc2bd54c77b79951919120d9cebe630e6d91c6497f16cd910f3a4d"} Jul 13 21:58:10.938228 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.938204 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" event={"ID":"a9b8b1b9-b95a-47bc-8f91-23012584c3e3","Type":"ContainerStarted","Data":"1d10968f854232d44154d7c4c45f7c10ea1006881c91bf82c71bf505684b185b"} Jul 13 21:58:10.939265 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.939242 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4m9ps" event={"ID":"f33a38a4-ac69-4c8f-915f-2e84fa701683","Type":"ContainerStarted","Data":"fbef3a4597f163cb576a0835174e49e65c9187c490746eac380987f1720fd6b7"} Jul 13 21:58:10.940272 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.940248 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-782jn" event={"ID":"a16c43cb-24ec-420c-953f-204d3584929c","Type":"ContainerStarted","Data":"1c397c5df14e8f275e5e1a8616e23fcea366197c8bc6d5c8027f43ab5747ca0f"} Jul 13 21:58:10.941284 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.941263 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwskc" event={"ID":"83271b6e-2653-44d9-8ddb-03bed0f66e39","Type":"ContainerStarted","Data":"81abbd329cf219db8cf54b462946e86d10c71ace2fdeee81cb8785af287ec6b1"} Jul 13 21:58:10.942397 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:10.942377 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" event={"ID":"5820bad8-3692-4171-b5e1-9f165718fe67","Type":"ContainerStarted","Data":"7a8f63ccac0b417b729342e27200674d1265c2fcb5af5b294d40778d6109d7e4"} Jul 13 21:58:11.396290 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:11.396253 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs\") pod \"network-metrics-daemon-tj476\" (UID: \"227fb8d9-c7c7-4722-bba7-f2c2fc9b482f\") " pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:11.396477 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:11.396402 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:58:11.396477 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:11.396468 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs podName:227fb8d9-c7c7-4722-bba7-f2c2fc9b482f nodeName:}" failed. No retries permitted until 2026-07-13 21:58:13.396448359 +0000 UTC m=+6.053653112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs") pod "network-metrics-daemon-tj476" (UID: "227fb8d9-c7c7-4722-bba7-f2c2fc9b482f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:58:11.497869 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:11.497092 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret\") pod \"global-pull-secret-syncer-qv2xs\" (UID: \"2b1adc79-d0de-44ab-b9c5-747c579b20cb\") " pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:11.497869 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:11.497139 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp64d\" (UniqueName: \"kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d\") pod \"network-check-target-qmrjq\" (UID: \"2ee46b9a-7019-44e4-a021-5c4ff5182a65\") " pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:11.497869 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:11.497391 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 21:58:11.497869 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:11.497421 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 21:58:11.497869 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:11.497440 2569 projected.go:196] Error preparing data for projected volume kube-api-access-xp64d for pod openshift-network-diagnostics/network-check-target-qmrjq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:58:11.497869 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:11.497495 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d podName:2ee46b9a-7019-44e4-a021-5c4ff5182a65 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:13.497477794 +0000 UTC m=+6.154682553 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xp64d" (UniqueName: "kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d") pod "network-check-target-qmrjq" (UID: "2ee46b9a-7019-44e4-a021-5c4ff5182a65") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:58:11.504704 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:11.498723 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 21:58:11.504704 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:11.498796 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret podName:2b1adc79-d0de-44ab-b9c5-747c579b20cb nodeName:}" failed. No retries permitted until 2026-07-13 21:58:13.498769817 +0000 UTC m=+6.155974569 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret") pod "global-pull-secret-syncer-qv2xs" (UID: "2b1adc79-d0de-44ab-b9c5-747c579b20cb") : object "kube-system"/"original-pull-secret" not registered Jul 13 21:58:11.507017 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:11.506995 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 21:58:11.813699 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:11.813568 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 21:53:09 +0000 UTC" deadline="2028-02-07 23:07:58.733717015 +0000 UTC" Jul 13 21:58:11.813699 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:11.813636 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13777h9m46.920093173s" Jul 13 21:58:11.930688 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:11.928308 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:11.930688 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:11.929798 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv2xs" podUID="2b1adc79-d0de-44ab-b9c5-747c579b20cb" Jul 13 21:58:11.930688 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:11.930388 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:11.930688 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:11.930476 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmrjq" podUID="2ee46b9a-7019-44e4-a021-5c4ff5182a65" Jul 13 21:58:11.930688 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:11.930557 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:11.930688 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:11.930643 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tj476" podUID="227fb8d9-c7c7-4722-bba7-f2c2fc9b482f" Jul 13 21:58:12.972688 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:12.971714 2569 generic.go:356] "Generic (PLEG): container finished" podID="161221439d14e33d352dfa94468aef89" containerID="8f16f3413e8d3111cd899e5870e2fa81d12f2b61d949c270bceb6fb2a0a69e4d" exitCode=0 Jul 13 21:58:12.972688 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:12.971814 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal" event={"ID":"161221439d14e33d352dfa94468aef89","Type":"ContainerDied","Data":"8f16f3413e8d3111cd899e5870e2fa81d12f2b61d949c270bceb6fb2a0a69e4d"} Jul 13 21:58:12.987410 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:12.986820 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-183.ec2.internal" event={"ID":"645e95dea7e9a87e47a32d256b790eae","Type":"ContainerStarted","Data":"d3de771173c5e531c6ce575279ca882d6a02d347394c5437e6c58491b80e08f6"} Jul 13 21:58:13.417859 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:13.417749 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs\") pod \"network-metrics-daemon-tj476\" (UID: \"227fb8d9-c7c7-4722-bba7-f2c2fc9b482f\") " pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:13.418032 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:13.417913 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:58:13.418032 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:13.417975 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs podName:227fb8d9-c7c7-4722-bba7-f2c2fc9b482f nodeName:}" failed. No retries permitted until 2026-07-13 21:58:17.417956738 +0000 UTC m=+10.075161490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs") pod "network-metrics-daemon-tj476" (UID: "227fb8d9-c7c7-4722-bba7-f2c2fc9b482f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:58:13.519838 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:13.519080 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret\") pod \"global-pull-secret-syncer-qv2xs\" (UID: \"2b1adc79-d0de-44ab-b9c5-747c579b20cb\") " pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:13.519838 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:13.519131 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp64d\" (UniqueName: \"kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d\") pod \"network-check-target-qmrjq\" (UID: \"2ee46b9a-7019-44e4-a021-5c4ff5182a65\") " pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:13.519838 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:13.519266 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 21:58:13.519838 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:13.519285 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 21:58:13.519838 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:13.519299 2569 projected.go:196] Error preparing data for projected volume kube-api-access-xp64d for pod openshift-network-diagnostics/network-check-target-qmrjq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:58:13.519838 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:13.519356 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d podName:2ee46b9a-7019-44e4-a021-5c4ff5182a65 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:17.51933706 +0000 UTC m=+10.176541828 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xp64d" (UniqueName: "kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d") pod "network-check-target-qmrjq" (UID: "2ee46b9a-7019-44e4-a021-5c4ff5182a65") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:58:13.519838 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:13.519748 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 21:58:13.519838 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:13.519798 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret podName:2b1adc79-d0de-44ab-b9c5-747c579b20cb nodeName:}" failed. No retries permitted until 2026-07-13 21:58:17.519783363 +0000 UTC m=+10.176988114 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret") pod "global-pull-secret-syncer-qv2xs" (UID: "2b1adc79-d0de-44ab-b9c5-747c579b20cb") : object "kube-system"/"original-pull-secret" not registered Jul 13 21:58:13.925328 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:13.925285 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:13.925328 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:13.925319 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:13.925609 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:13.925285 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:13.925609 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:13.925418 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmrjq" podUID="2ee46b9a-7019-44e4-a021-5c4ff5182a65" Jul 13 21:58:13.925609 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:13.925527 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv2xs" podUID="2b1adc79-d0de-44ab-b9c5-747c579b20cb" Jul 13 21:58:13.925771 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:13.925633 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tj476" podUID="227fb8d9-c7c7-4722-bba7-f2c2fc9b482f" Jul 13 21:58:15.926580 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:15.925683 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:15.926580 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:15.925816 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv2xs" podUID="2b1adc79-d0de-44ab-b9c5-747c579b20cb" Jul 13 21:58:15.926580 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:15.926272 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:15.926580 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:15.926366 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmrjq" podUID="2ee46b9a-7019-44e4-a021-5c4ff5182a65" Jul 13 21:58:15.926580 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:15.926437 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:15.926580 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:15.926518 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tj476" podUID="227fb8d9-c7c7-4722-bba7-f2c2fc9b482f" Jul 13 21:58:17.453083 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:17.453045 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs\") pod \"network-metrics-daemon-tj476\" (UID: \"227fb8d9-c7c7-4722-bba7-f2c2fc9b482f\") " pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:17.453648 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:17.453240 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:58:17.453648 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:17.453306 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs podName:227fb8d9-c7c7-4722-bba7-f2c2fc9b482f nodeName:}" failed. No retries permitted until 2026-07-13 21:58:25.453286724 +0000 UTC m=+18.110491473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs") pod "network-metrics-daemon-tj476" (UID: "227fb8d9-c7c7-4722-bba7-f2c2fc9b482f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:58:17.554086 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:17.554047 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret\") pod \"global-pull-secret-syncer-qv2xs\" (UID: \"2b1adc79-d0de-44ab-b9c5-747c579b20cb\") " pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:17.554241 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:17.554115 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp64d\" (UniqueName: \"kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d\") pod \"network-check-target-qmrjq\" (UID: \"2ee46b9a-7019-44e4-a021-5c4ff5182a65\") " pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:17.554241 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:17.554219 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 21:58:17.554368 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:17.554292 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret podName:2b1adc79-d0de-44ab-b9c5-747c579b20cb nodeName:}" failed. No retries permitted until 2026-07-13 21:58:25.554273343 +0000 UTC m=+18.211478092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret") pod "global-pull-secret-syncer-qv2xs" (UID: "2b1adc79-d0de-44ab-b9c5-747c579b20cb") : object "kube-system"/"original-pull-secret" not registered Jul 13 21:58:17.554368 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:17.554309 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 21:58:17.554368 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:17.554332 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 21:58:17.554368 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:17.554349 2569 projected.go:196] Error preparing data for projected volume kube-api-access-xp64d for pod openshift-network-diagnostics/network-check-target-qmrjq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:58:17.554585 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:17.554408 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d podName:2ee46b9a-7019-44e4-a021-5c4ff5182a65 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:25.554391522 +0000 UTC m=+18.211596271 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xp64d" (UniqueName: "kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d") pod "network-check-target-qmrjq" (UID: "2ee46b9a-7019-44e4-a021-5c4ff5182a65") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:58:17.925316 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:17.925218 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:17.925316 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:17.925255 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:17.925528 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:17.925335 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmrjq" podUID="2ee46b9a-7019-44e4-a021-5c4ff5182a65" Jul 13 21:58:17.925528 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:17.925221 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:17.925528 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:17.925484 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv2xs" podUID="2b1adc79-d0de-44ab-b9c5-747c579b20cb" Jul 13 21:58:17.926574 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:17.926500 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tj476" podUID="227fb8d9-c7c7-4722-bba7-f2c2fc9b482f" Jul 13 21:58:19.925342 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:19.925265 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:19.925342 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:19.925280 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:19.925872 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:19.925402 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv2xs" podUID="2b1adc79-d0de-44ab-b9c5-747c579b20cb" Jul 13 21:58:19.925872 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:19.925457 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:19.925872 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:19.925587 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmrjq" podUID="2ee46b9a-7019-44e4-a021-5c4ff5182a65" Jul 13 21:58:19.925872 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:19.925685 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tj476" podUID="227fb8d9-c7c7-4722-bba7-f2c2fc9b482f" Jul 13 21:58:21.924595 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:21.924504 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:21.925002 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:21.924504 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:21.925002 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:21.924649 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv2xs" podUID="2b1adc79-d0de-44ab-b9c5-747c579b20cb" Jul 13 21:58:21.925002 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:21.924713 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tj476" podUID="227fb8d9-c7c7-4722-bba7-f2c2fc9b482f" Jul 13 21:58:21.925002 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:21.924736 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:21.925002 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:21.924802 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmrjq" podUID="2ee46b9a-7019-44e4-a021-5c4ff5182a65" Jul 13 21:58:23.925163 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:23.925067 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:23.925163 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:23.925092 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:23.925698 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:23.925198 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:23.925698 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:23.925194 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmrjq" podUID="2ee46b9a-7019-44e4-a021-5c4ff5182a65" Jul 13 21:58:23.925698 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:23.925307 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tj476" podUID="227fb8d9-c7c7-4722-bba7-f2c2fc9b482f" Jul 13 21:58:23.925698 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:23.925453 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv2xs" podUID="2b1adc79-d0de-44ab-b9c5-747c579b20cb" Jul 13 21:58:25.515592 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:25.515557 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs\") pod \"network-metrics-daemon-tj476\" (UID: \"227fb8d9-c7c7-4722-bba7-f2c2fc9b482f\") " pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:25.516105 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:25.515700 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:58:25.516105 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:25.515780 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs podName:227fb8d9-c7c7-4722-bba7-f2c2fc9b482f nodeName:}" failed. No retries permitted until 2026-07-13 21:58:41.51575618 +0000 UTC m=+34.172960929 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs") pod "network-metrics-daemon-tj476" (UID: "227fb8d9-c7c7-4722-bba7-f2c2fc9b482f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:58:25.616424 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:25.616390 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret\") pod \"global-pull-secret-syncer-qv2xs\" (UID: \"2b1adc79-d0de-44ab-b9c5-747c579b20cb\") " pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:25.616424 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:25.616431 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp64d\" (UniqueName: \"kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d\") pod \"network-check-target-qmrjq\" (UID: \"2ee46b9a-7019-44e4-a021-5c4ff5182a65\") " pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:25.616706 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:25.616561 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 21:58:25.616706 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:25.616562 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 21:58:25.616706 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:25.616576 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 21:58:25.616706 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:25.616594 2569 projected.go:196] Error preparing data for projected volume kube-api-access-xp64d for pod openshift-network-diagnostics/network-check-target-qmrjq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:58:25.616706 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:25.616640 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret podName:2b1adc79-d0de-44ab-b9c5-747c579b20cb nodeName:}" failed. No retries permitted until 2026-07-13 21:58:41.616619089 +0000 UTC m=+34.273823855 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret") pod "global-pull-secret-syncer-qv2xs" (UID: "2b1adc79-d0de-44ab-b9c5-747c579b20cb") : object "kube-system"/"original-pull-secret" not registered Jul 13 21:58:25.616706 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:25.616658 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d podName:2ee46b9a-7019-44e4-a021-5c4ff5182a65 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:41.616649988 +0000 UTC m=+34.273854734 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xp64d" (UniqueName: "kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d") pod "network-check-target-qmrjq" (UID: "2ee46b9a-7019-44e4-a021-5c4ff5182a65") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:58:25.925406 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:25.925317 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:25.925406 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:25.925354 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:25.925682 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:25.925317 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:25.925682 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:25.925454 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmrjq" podUID="2ee46b9a-7019-44e4-a021-5c4ff5182a65" Jul 13 21:58:25.925682 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:25.925559 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv2xs" podUID="2b1adc79-d0de-44ab-b9c5-747c579b20cb" Jul 13 21:58:25.925682 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:25.925663 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tj476" podUID="227fb8d9-c7c7-4722-bba7-f2c2fc9b482f" Jul 13 21:58:27.925801 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:27.925765 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:27.926254 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:27.925858 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:27.926254 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:27.925880 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:27.926254 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:27.925872 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmrjq" podUID="2ee46b9a-7019-44e4-a021-5c4ff5182a65" Jul 13 21:58:27.926254 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:27.925968 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tj476" podUID="227fb8d9-c7c7-4722-bba7-f2c2fc9b482f" Jul 13 21:58:27.926254 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:27.926059 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv2xs" podUID="2b1adc79-d0de-44ab-b9c5-747c579b20cb" Jul 13 21:58:29.925739 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:29.925557 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:29.926350 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:29.925551 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:29.926350 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:29.925828 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmrjq" podUID="2ee46b9a-7019-44e4-a021-5c4ff5182a65" Jul 13 21:58:29.926350 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:29.925910 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tj476" podUID="227fb8d9-c7c7-4722-bba7-f2c2fc9b482f" Jul 13 21:58:29.926350 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:29.925685 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:29.926350 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:29.925992 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv2xs" podUID="2b1adc79-d0de-44ab-b9c5-747c579b20cb" Jul 13 21:58:30.015430 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:30.015396 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vpg2f" event={"ID":"3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0","Type":"ContainerStarted","Data":"49d5d465e70a0067cfec49b6bd1cae52f7c46b956cf0dac57f741b99b22ad0b5"} Jul 13 21:58:30.016941 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:30.016907 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rvqz8" event={"ID":"eab93415-1e38-4dbe-9c23-dbc2b0354a12","Type":"ContainerStarted","Data":"4690b7c580a28a0dcc27936f77662f733c426e1670b2f697f48f95a4453c2fd8"} Jul 13 21:58:30.018401 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:30.018371 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" event={"ID":"a9b8b1b9-b95a-47bc-8f91-23012584c3e3","Type":"ContainerStarted","Data":"fc68630ce42dc73315d50db136f01168aab07441402eeb8fd4e914d1847acead"} Jul 13 21:58:30.020257 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:30.020236 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal" event={"ID":"161221439d14e33d352dfa94468aef89","Type":"ContainerStarted","Data":"da1f777a99451864cd7bc13da11c7c4f83b2f940588437efc9ec1dfb059cfb6a"} Jul 13 21:58:30.021636 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:30.021615 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4m9ps" event={"ID":"f33a38a4-ac69-4c8f-915f-2e84fa701683","Type":"ContainerStarted","Data":"cee950742fa7d649805d78e7f4e3f3dd358bb188d1a0db0327bf8ce56dd46b1b"} Jul 13 21:58:30.022973 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:30.022947 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-782jn" event={"ID":"a16c43cb-24ec-420c-953f-204d3584929c","Type":"ContainerStarted","Data":"a29fe58b673e179c563fb80554f10cd60bd8334c7a46470e96f02878c7e80090"} Jul 13 21:58:30.024435 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:30.024408 2569 generic.go:356] "Generic (PLEG): container finished" podID="83271b6e-2653-44d9-8ddb-03bed0f66e39" containerID="6573370c755a49d06f34dc8b29443cdf8f3c5477e5d61c87794f4e3b1ab44dc8" exitCode=0 Jul 13 21:58:30.024562 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:30.024485 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwskc" event={"ID":"83271b6e-2653-44d9-8ddb-03bed0f66e39","Type":"ContainerDied","Data":"6573370c755a49d06f34dc8b29443cdf8f3c5477e5d61c87794f4e3b1ab44dc8"} Jul 13 21:58:30.026132 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:30.026108 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" event={"ID":"5820bad8-3692-4171-b5e1-9f165718fe67","Type":"ContainerStarted","Data":"2e0f0e85907278ca0d38e2e55eee9e1fa49b424ae17afccac607cf439dae3ece"} Jul 13 21:58:30.030174 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:30.030102 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vpg2f" podStartSLOduration=2.61642359 podStartE2EDuration="22.030091914s" totalImagesPullingTime="19.413668324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:30.029935437 +0000 UTC m=+22.687140208" watchObservedRunningTime="2026-07-13 21:58:30.030091914 +0000 UTC m=+22.687296682" Jul 13 21:58:30.030283 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:30.030172 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-183.ec2.internal" podStartSLOduration=21.030167625 podStartE2EDuration="21.030167625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:13.004528102 +0000 UTC m=+5.661732871" watchObservedRunningTime="2026-07-13 21:58:30.030167625 +0000 UTC m=+22.687372390" Jul 13 21:58:30.045182 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:30.045143 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4m9ps" podStartSLOduration=2.577323086 podStartE2EDuration="22.045131781s" totalImagesPullingTime="19.467808695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:30.045014491 +0000 UTC m=+22.702219259" watchObservedRunningTime="2026-07-13 21:58:30.045131781 +0000 UTC m=+22.702336548" Jul 13 21:58:30.063787 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:30.063746 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-183.ec2.internal" podStartSLOduration=21.063734872 podStartE2EDuration="21.063734872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:30.063723228 +0000 UTC m=+22.720928000" watchObservedRunningTime="2026-07-13 21:58:30.063734872 +0000 UTC m=+22.720939640" Jul 13 21:58:30.104881 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:30.104828 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rvqz8" podStartSLOduration=2.580850473 podStartE2EDuration="22.104807137s" totalImagesPullingTime="19.523956664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:30.104456573 +0000 UTC m=+22.761661339" watchObservedRunningTime="2026-07-13 21:58:30.104807137 +0000 UTC m=+22.762011906" Jul 13 21:58:30.122033 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:30.121997 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n9gc7" podStartSLOduration=3.602700952 podStartE2EDuration="23.121985212s" totalImagesPullingTime="19.51928426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:30.121300686 +0000 UTC m=+22.778505454" watchObservedRunningTime="2026-07-13 21:58:30.121985212 +0000 UTC m=+22.779189980" Jul 13 21:58:30.137448 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:30.137274 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-782jn" podStartSLOduration=2.6890533359999997 podStartE2EDuration="22.137264961s" totalImagesPullingTime="19.448211625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:30.136852005 +0000 UTC m=+22.794056773" watchObservedRunningTime="2026-07-13 21:58:30.137264961 +0000 UTC m=+22.794469726" Jul 13 21:58:31.030119 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:31.030079 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g4fk2" event={"ID":"7340cc2c-3aa2-4811-916f-90f9e4b30ae3","Type":"ContainerStarted","Data":"fbc78b6ee11c12c85b16614bfe94a93b4780fce2b262728d2b025221ca86f5ec"} Jul 13 21:58:31.035666 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:31.035643 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 21:58:31.036000 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:31.035972 2569 generic.go:356] "Generic (PLEG): container finished" podID="d11fbab3-afd4-4eaa-9b10-c0262d17fc6d" containerID="43b0c67e8d8c992f9ac53964f35ef9579c7d23b650e16b2dc4b9a51875f5925e" exitCode=1 Jul 13 21:58:31.036113 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:31.036048 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" event={"ID":"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d","Type":"ContainerStarted","Data":"e47824213a9ceaa2368118e5c667ed6b5f231515688236da736c7c3dd0d02f04"} Jul 13 21:58:31.036176 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:31.036123 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" event={"ID":"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d","Type":"ContainerStarted","Data":"c81f95cd95d237403d0946b5d871fa0b9d101d2ffb899ded695ab4adb76d88b4"} Jul 13 21:58:31.036227 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:31.036185 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" event={"ID":"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d","Type":"ContainerStarted","Data":"cce20d7c99246a2d3d78ed76f09e85370746328460233efc6643bf9136615ff9"} Jul 13 21:58:31.036227 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:31.036221 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" event={"ID":"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d","Type":"ContainerStarted","Data":"f0d21dae8fed2c568ac1108132bced2764e139e01458a44ea63a6c88b47adbde"} Jul 13 21:58:31.036324 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:31.036250 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" event={"ID":"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d","Type":"ContainerDied","Data":"43b0c67e8d8c992f9ac53964f35ef9579c7d23b650e16b2dc4b9a51875f5925e"} Jul 13 21:58:31.036324 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:31.036280 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" event={"ID":"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d","Type":"ContainerStarted","Data":"395715565ec8cf73e940942da1c4af655aaec3045e26f4c4440b28accd57d428"} Jul 13 21:58:31.045447 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:31.045399 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g4fk2" podStartSLOduration=3.553378715 podStartE2EDuration="23.045381621s" totalImagesPullingTime="19.492002906s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:31.044524141 +0000 UTC m=+23.701728909" watchObservedRunningTime="2026-07-13 21:58:31.045381621 +0000 UTC m=+23.702586390" Jul 13 21:58:31.170443 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:31.170276 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 13 21:58:31.847361 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:31.847246 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T21:58:31.170423868Z","UUID":"6d5c2de6-c805-4a2f-af47-955b2b43a1a4","Handler":null,"Name":"","Endpoint":""} Jul 13 21:58:31.849242 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:31.849223 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 13 21:58:31.849379 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:31.849265 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 13 21:58:31.925378 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:31.925334 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:31.925378 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:31.925382 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:31.925653 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:31.925480 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:31.925653 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:31.925495 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv2xs" podUID="2b1adc79-d0de-44ab-b9c5-747c579b20cb" Jul 13 21:58:31.925758 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:31.925647 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tj476" podUID="227fb8d9-c7c7-4722-bba7-f2c2fc9b482f" Jul 13 21:58:31.925813 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:31.925745 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmrjq" podUID="2ee46b9a-7019-44e4-a021-5c4ff5182a65" Jul 13 21:58:32.039946 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:32.039910 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" event={"ID":"5820bad8-3692-4171-b5e1-9f165718fe67","Type":"ContainerStarted","Data":"d40de2863bbcbb38cb7def382bdceb80c1bc5939afc3af4c1da51e7ad636b265"} Jul 13 21:58:33.044719 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:33.044678 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" event={"ID":"5820bad8-3692-4171-b5e1-9f165718fe67","Type":"ContainerStarted","Data":"73f91cec48d1139321867088ab9f3419a1970a5ce9af9f2082938fe9919975e8"} Jul 13 21:58:33.064175 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:33.064123 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hcrs2" podStartSLOduration=4.598781731 podStartE2EDuration="26.064107403s" totalImagesPullingTime="21.465325672s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:58:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:33.063827033 +0000 UTC m=+25.721031813" watchObservedRunningTime="2026-07-13 21:58:33.064107403 +0000 UTC m=+25.721312172" Jul 13 21:58:33.661865 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:33.661826 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vpg2f" Jul 13 21:58:33.662717 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:33.662691 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vpg2f" Jul 13 21:58:33.924791 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:33.924701 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:33.924791 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:33.924734 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:33.925000 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:33.924830 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmrjq" podUID="2ee46b9a-7019-44e4-a021-5c4ff5182a65" Jul 13 21:58:33.925000 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:33.924892 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tj476" podUID="227fb8d9-c7c7-4722-bba7-f2c2fc9b482f" Jul 13 21:58:33.925000 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:33.924946 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:33.925138 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:33.925020 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv2xs" podUID="2b1adc79-d0de-44ab-b9c5-747c579b20cb" Jul 13 21:58:34.047119 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:34.047087 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vpg2f" Jul 13 21:58:34.047825 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:34.047806 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vpg2f" Jul 13 21:58:35.050258 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:35.050222 2569 generic.go:356] "Generic (PLEG): container finished" podID="83271b6e-2653-44d9-8ddb-03bed0f66e39" containerID="85690ec59f4be2b64db4bb3660215ffa8cb465e77ff1b11c671c9408e514aa56" exitCode=0 Jul 13 21:58:35.050934 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:35.050305 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwskc" event={"ID":"83271b6e-2653-44d9-8ddb-03bed0f66e39","Type":"ContainerDied","Data":"85690ec59f4be2b64db4bb3660215ffa8cb465e77ff1b11c671c9408e514aa56"} Jul 13 21:58:35.053511 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:35.053498 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 21:58:35.053969 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:35.053949 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" event={"ID":"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d","Type":"ContainerStarted","Data":"44fc5b40497a1bef98e06f96248ac5af0ff0a8f6b304df0e96e01eae5539a7e4"} Jul 13 21:58:35.925546 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:35.925363 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:35.925702 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:35.925375 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:35.925702 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:35.925622 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmrjq" podUID="2ee46b9a-7019-44e4-a021-5c4ff5182a65" Jul 13 21:58:35.925702 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:35.925428 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:35.925860 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:35.925697 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tj476" podUID="227fb8d9-c7c7-4722-bba7-f2c2fc9b482f" Jul 13 21:58:35.925860 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:35.925807 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv2xs" podUID="2b1adc79-d0de-44ab-b9c5-747c579b20cb" Jul 13 21:58:37.061476 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:37.061450 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 21:58:37.061925 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:37.061865 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" event={"ID":"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d","Type":"ContainerStarted","Data":"966c5d2ca09ac6e32e6791fbc692f000a2801d8742e01550a0c52c9fe535715f"} Jul 13 21:58:37.062087 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:37.062066 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:37.062311 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:37.062298 2569 scope.go:117] "RemoveContainer" containerID="43b0c67e8d8c992f9ac53964f35ef9579c7d23b650e16b2dc4b9a51875f5925e" Jul 13 21:58:37.064198 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:37.064177 2569 generic.go:356] "Generic (PLEG): container finished" podID="83271b6e-2653-44d9-8ddb-03bed0f66e39" containerID="07c1272741a92115cf507d6d96ec3ff2474ad6ff3125ee00fa96bc5132750a93" exitCode=0 Jul 13 21:58:37.064274 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:37.064209 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwskc" event={"ID":"83271b6e-2653-44d9-8ddb-03bed0f66e39","Type":"ContainerDied","Data":"07c1272741a92115cf507d6d96ec3ff2474ad6ff3125ee00fa96bc5132750a93"} Jul 13 21:58:37.079073 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:37.079051 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:37.925424 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:37.925394 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:37.925586 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:37.925475 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:37.925586 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:37.925480 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmrjq" podUID="2ee46b9a-7019-44e4-a021-5c4ff5182a65" Jul 13 21:58:37.925668 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:37.925616 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tj476" podUID="227fb8d9-c7c7-4722-bba7-f2c2fc9b482f" Jul 13 21:58:37.925668 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:37.925651 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:37.925748 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:37.925731 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv2xs" podUID="2b1adc79-d0de-44ab-b9c5-747c579b20cb" Jul 13 21:58:38.071678 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:38.071649 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwskc" event={"ID":"83271b6e-2653-44d9-8ddb-03bed0f66e39","Type":"ContainerStarted","Data":"4a68c7cec87eb5ce84652a286061665091973ae75602a8c4b7cea1abf32de3b7"} Jul 13 21:58:38.075612 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:38.075594 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 21:58:38.075947 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:38.075920 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" event={"ID":"d11fbab3-afd4-4eaa-9b10-c0262d17fc6d","Type":"ContainerStarted","Data":"0b95ba62f1c3401780ceda4167a858d94c8ca746b8c119374617cafc052d2c31"} Jul 13 21:58:38.076133 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:38.076113 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:38.076220 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:38.076168 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:38.090079 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:38.090056 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:58:38.125840 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:38.125786 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" podStartSLOduration=10.290274041 podStartE2EDuration="30.125766439s" totalImagesPullingTime="19.835492398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:38.125033699 +0000 UTC m=+30.782238469" watchObservedRunningTime="2026-07-13 21:58:38.125766439 +0000 UTC m=+30.782971211" Jul 13 21:58:38.375072 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:38.375041 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tj476"] Jul 13 21:58:38.375216 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:38.375166 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:38.375302 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:38.375280 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tj476" podUID="227fb8d9-c7c7-4722-bba7-f2c2fc9b482f" Jul 13 21:58:38.382074 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:38.382047 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qv2xs"] Jul 13 21:58:38.382206 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:38.382135 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:38.382266 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:38.382224 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv2xs" podUID="2b1adc79-d0de-44ab-b9c5-747c579b20cb" Jul 13 21:58:38.382979 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:38.382813 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qmrjq"] Jul 13 21:58:38.382979 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:38.382899 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:38.383148 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:38.382987 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmrjq" podUID="2ee46b9a-7019-44e4-a021-5c4ff5182a65" Jul 13 21:58:39.081325 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:39.081296 2569 generic.go:356] "Generic (PLEG): container finished" podID="83271b6e-2653-44d9-8ddb-03bed0f66e39" containerID="4a68c7cec87eb5ce84652a286061665091973ae75602a8c4b7cea1abf32de3b7" exitCode=0 Jul 13 21:58:39.082088 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:39.081374 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwskc" event={"ID":"83271b6e-2653-44d9-8ddb-03bed0f66e39","Type":"ContainerDied","Data":"4a68c7cec87eb5ce84652a286061665091973ae75602a8c4b7cea1abf32de3b7"} Jul 13 21:58:39.924754 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:39.924720 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:39.924925 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:39.924757 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:39.924925 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:39.924737 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:39.924925 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:39.924859 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tj476" podUID="227fb8d9-c7c7-4722-bba7-f2c2fc9b482f" Jul 13 21:58:39.925078 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:39.924957 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv2xs" podUID="2b1adc79-d0de-44ab-b9c5-747c579b20cb" Jul 13 21:58:39.925078 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:39.925045 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmrjq" podUID="2ee46b9a-7019-44e4-a021-5c4ff5182a65" Jul 13 21:58:41.535617 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:41.535578 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs\") pod \"network-metrics-daemon-tj476\" (UID: \"227fb8d9-c7c7-4722-bba7-f2c2fc9b482f\") " pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:41.536098 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:41.535716 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:58:41.536098 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:41.535776 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs podName:227fb8d9-c7c7-4722-bba7-f2c2fc9b482f nodeName:}" failed. No retries permitted until 2026-07-13 21:59:13.535761102 +0000 UTC m=+66.192965864 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs") pod "network-metrics-daemon-tj476" (UID: "227fb8d9-c7c7-4722-bba7-f2c2fc9b482f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:58:41.635964 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:41.635926 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp64d\" (UniqueName: \"kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d\") pod \"network-check-target-qmrjq\" (UID: \"2ee46b9a-7019-44e4-a021-5c4ff5182a65\") " pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:41.636129 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:41.636022 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret\") pod \"global-pull-secret-syncer-qv2xs\" (UID: \"2b1adc79-d0de-44ab-b9c5-747c579b20cb\") " pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:41.636129 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:41.636096 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 21:58:41.636129 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:41.636117 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 21:58:41.636129 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:41.636119 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 21:58:41.636129 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:41.636128 2569 projected.go:196] Error preparing data for projected volume kube-api-access-xp64d for pod openshift-network-diagnostics/network-check-target-qmrjq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:58:41.636334 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:41.636180 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret podName:2b1adc79-d0de-44ab-b9c5-747c579b20cb nodeName:}" failed. No retries permitted until 2026-07-13 21:59:13.636162849 +0000 UTC m=+66.293367602 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret") pod "global-pull-secret-syncer-qv2xs" (UID: "2b1adc79-d0de-44ab-b9c5-747c579b20cb") : object "kube-system"/"original-pull-secret" not registered Jul 13 21:58:41.636334 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:41.636201 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d podName:2ee46b9a-7019-44e4-a021-5c4ff5182a65 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:13.636190391 +0000 UTC m=+66.293395144 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xp64d" (UniqueName: "kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d") pod "network-check-target-qmrjq" (UID: "2ee46b9a-7019-44e4-a021-5c4ff5182a65") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:58:41.925148 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:41.925034 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:41.925148 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:41.925102 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:41.925370 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:41.925197 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qv2xs" podUID="2b1adc79-d0de-44ab-b9c5-747c579b20cb" Jul 13 21:58:41.925370 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:41.925277 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tj476" podUID="227fb8d9-c7c7-4722-bba7-f2c2fc9b482f" Jul 13 21:58:41.925370 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:41.925313 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:41.925508 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:41.925422 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qmrjq" podUID="2ee46b9a-7019-44e4-a021-5c4ff5182a65" Jul 13 21:58:42.578201 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.578031 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-183.ec2.internal" event="NodeReady" Jul 13 21:58:42.578491 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.578291 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 13 21:58:42.615414 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.615396 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-nj7h6"] Jul 13 21:58:42.630289 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.630270 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-sh5jg"] Jul 13 21:58:42.630418 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.630401 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-nj7h6" Jul 13 21:58:42.636640 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.636342 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 13 21:58:42.636640 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.636407 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-f4fwc\"" Jul 13 21:58:42.636640 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.636420 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 13 21:58:42.636640 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.636426 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 13 21:58:42.636640 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.636355 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 13 21:58:42.642789 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.642608 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 13 21:58:42.646390 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.646310 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f784dd4c9-l4qbc"] Jul 13 21:58:42.647034 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.646602 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-sh5jg" Jul 13 21:58:42.649326 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.649304 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-lxvdn\"" Jul 13 21:58:42.650356 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.650340 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 13 21:58:42.650703 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.650686 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 13 21:58:42.666416 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.666392 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk"] Jul 13 21:58:42.666558 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.666520 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.670460 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.670442 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 13 21:58:42.675890 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.675875 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p9jhm\"" Jul 13 21:58:42.675967 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.675906 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 13 21:58:42.682765 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.682750 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 13 21:58:42.690618 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.690603 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g4ds9"] Jul 13 21:58:42.690714 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.690701 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk" Jul 13 21:58:42.695065 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.695048 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 13 21:58:42.698387 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.698370 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ccc6k\"" Jul 13 21:58:42.698892 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.698878 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 13 21:58:42.698979 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.698910 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 13 21:58:42.699463 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.699449 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 13 21:58:42.714510 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.714493 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr"] Jul 13 21:58:42.714625 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.714611 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g4ds9" Jul 13 21:58:42.721678 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.721661 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 13 21:58:42.722028 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.722015 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8z79n\"" Jul 13 21:58:42.722588 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.722572 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 13 21:58:42.723496 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.723019 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 13 21:58:42.726021 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.726004 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 13 21:58:42.734700 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.734656 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-nj7h6"] Jul 13 21:58:42.734700 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.734683 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g4ds9"] Jul 13 21:58:42.734700 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.734698 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr"] Jul 13 21:58:42.734841 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.734719 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-sh5jg"] Jul 13 21:58:42.734841 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.734735 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-85gwn"] Jul 13 21:58:42.734841 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.734796 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" Jul 13 21:58:42.740069 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.740048 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 13 21:58:42.740260 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.740248 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9vd8k\"" Jul 13 21:58:42.741218 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.741204 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 13 21:58:42.742209 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.742193 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 13 21:58:42.744425 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.744409 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 13 21:58:42.745585 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.745568 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a5e8dab5-94da-46d0-8653-bcc21e849fc1-config\") pod \"console-operator-7866497494-nj7h6\" (UID: \"a5e8dab5-94da-46d0-8653-bcc21e849fc1\") " pod="openshift-console-operator/console-operator-7866497494-nj7h6" Jul 13 21:58:42.745649 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.745593 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a5e8dab5-94da-46d0-8653-bcc21e849fc1-serving-cert\") pod \"console-operator-7866497494-nj7h6\" (UID: \"a5e8dab5-94da-46d0-8653-bcc21e849fc1\") " pod="openshift-console-operator/console-operator-7866497494-nj7h6" Jul 13 21:58:42.745649 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.745613 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndl6c\" (UniqueName: \"kubernetes.io/projected/a5e8dab5-94da-46d0-8653-bcc21e849fc1-kube-api-access-ndl6c\") pod \"console-operator-7866497494-nj7h6\" (UID: \"a5e8dab5-94da-46d0-8653-bcc21e849fc1\") " pod="openshift-console-operator/console-operator-7866497494-nj7h6" Jul 13 21:58:42.745717 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.745645 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpbf6\" (UniqueName: \"kubernetes.io/projected/2b79086d-97c9-4fe0-a960-a44518765dcf-kube-api-access-xpbf6\") pod \"volume-data-source-validator-78495597f6-sh5jg\" (UID: \"2b79086d-97c9-4fe0-a960-a44518765dcf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-sh5jg" Jul 13 21:58:42.745717 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.745703 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4c27\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-kube-api-access-b4c27\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.745780 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.745723 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5e8dab5-94da-46d0-8653-bcc21e849fc1-trusted-ca\") pod \"console-operator-7866497494-nj7h6\" (UID: \"a5e8dab5-94da-46d0-8653-bcc21e849fc1\") " pod="openshift-console-operator/console-operator-7866497494-nj7h6" Jul 13 21:58:42.745780 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.745745 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6cac02d-f5da-4569-894c-0328af666e63-trusted-ca\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.745780 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.745767 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6cac02d-f5da-4569-894c-0328af666e63-registry-certificates\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.745863 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.745781 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6cac02d-f5da-4569-894c-0328af666e63-installation-pull-secrets\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.745863 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.745829 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6cac02d-f5da-4569-894c-0328af666e63-image-registry-private-configuration\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.745863 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.745845 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6cac02d-f5da-4569-894c-0328af666e63-ca-trust-extracted\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.745951 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.745902 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.745951 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.745921 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-bound-sa-token\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.752764 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.752745 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-n55wp"] Jul 13 21:58:42.752872 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.752858 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:42.755863 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.755846 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 13 21:58:42.755960 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.755878 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 13 21:58:42.755960 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.755882 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 13 21:58:42.756557 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.756521 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 13 21:58:42.760910 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.760892 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4dt4r\"" Jul 13 21:58:42.763658 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.763645 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9dkjb"] Jul 13 21:58:42.763754 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.763741 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-n55wp" Jul 13 21:58:42.769145 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.769116 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-987r4\"" Jul 13 21:58:42.769579 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.769560 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 21:58:42.769953 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.769935 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 21:58:42.770180 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.770154 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 13 21:58:42.778963 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.778946 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zghwn"] Jul 13 21:58:42.779062 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.779048 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9dkjb" Jul 13 21:58:42.781364 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.781343 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jqfwq\"" Jul 13 21:58:42.781895 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.781876 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 13 21:58:42.781975 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.781878 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 13 21:58:42.791023 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.790928 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-768d869684-xh5gb"] Jul 13 21:58:42.791147 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.791133 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zghwn" Jul 13 21:58:42.793844 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.793827 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-l8qj7\"" Jul 13 21:58:42.793950 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.793934 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 13 21:58:42.794151 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.794136 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 13 21:58:42.794261 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.794246 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 13 21:58:42.794261 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.794256 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 13 21:58:42.805189 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.805175 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f784dd4c9-l4qbc"] Jul 13 21:58:42.805261 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.805199 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zghwn"] Jul 13 21:58:42.805261 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.805213 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9dkjb"] Jul 13 21:58:42.805261 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.805231 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-n55wp"] Jul 13 21:58:42.805261 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.805243 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk"] Jul 13 21:58:42.805261 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.805257 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-85gwn"] Jul 13 21:58:42.805445 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.805268 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-768d869684-xh5gb"] Jul 13 21:58:42.805445 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.805269 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:42.805445 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.805282 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-thphz"] Jul 13 21:58:42.808172 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.808150 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 13 21:58:42.808390 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.808377 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-l7jtp\"" Jul 13 21:58:42.808946 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.808931 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 13 21:58:42.808946 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.808944 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 13 21:58:42.810437 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.810279 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 13 21:58:42.810437 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.810295 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 13 21:58:42.810712 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.810681 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 13 21:58:42.819274 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.819258 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-thphz"] Jul 13 21:58:42.819351 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.819326 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-thphz" Jul 13 21:58:42.821950 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.821929 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 13 21:58:42.821950 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.821951 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 13 21:58:42.822115 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.821995 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7pv2b\"" Jul 13 21:58:42.822115 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.822035 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 13 21:58:42.846121 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846102 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndl6c\" (UniqueName: \"kubernetes.io/projected/a5e8dab5-94da-46d0-8653-bcc21e849fc1-kube-api-access-ndl6c\") pod \"console-operator-7866497494-nj7h6\" (UID: \"a5e8dab5-94da-46d0-8653-bcc21e849fc1\") " pod="openshift-console-operator/console-operator-7866497494-nj7h6" Jul 13 21:58:42.846217 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846130 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxlfm\" (UniqueName: \"kubernetes.io/projected/2c6dc74b-69a0-453a-94be-8f5006659d1e-kube-api-access-gxlfm\") pod \"network-check-source-869dcfcdcd-n55wp\" (UID: \"2c6dc74b-69a0-453a-94be-8f5006659d1e\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-n55wp" Jul 13 21:58:42.846217 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846148 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfj2z\" (UniqueName: \"kubernetes.io/projected/30ccc289-8e87-46a2-80b7-064bb0724db7-kube-api-access-sfj2z\") pod \"cluster-samples-operator-65db858894-mscwk\" (UID: \"30ccc289-8e87-46a2-80b7-064bb0724db7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk" Jul 13 21:58:42.846217 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846164 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpbf6\" (UniqueName: \"kubernetes.io/projected/2b79086d-97c9-4fe0-a960-a44518765dcf-kube-api-access-xpbf6\") pod \"volume-data-source-validator-78495597f6-sh5jg\" (UID: \"2b79086d-97c9-4fe0-a960-a44518765dcf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-sh5jg" Jul 13 21:58:42.846217 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846181 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a59df9a-555c-4676-9b35-d1f8b2752e82-tmp\") pod \"insights-operator-68d9dd94cf-85gwn\" (UID: \"3a59df9a-555c-4676-9b35-d1f8b2752e82\") " pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:42.846401 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846231 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6cac02d-f5da-4569-894c-0328af666e63-trusted-ca\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.846401 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846266 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmzk2\" (UniqueName: \"kubernetes.io/projected/80fea354-ee7a-49af-8201-0bc32959c2d0-kube-api-access-tmzk2\") pod \"service-ca-operator-7695fc6c8c-g4ds9\" (UID: \"80fea354-ee7a-49af-8201-0bc32959c2d0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g4ds9" Jul 13 21:58:42.846401 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846372 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6cac02d-f5da-4569-894c-0328af666e63-registry-certificates\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.846556 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846404 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a5e8dab5-94da-46d0-8653-bcc21e849fc1-config\") pod \"console-operator-7866497494-nj7h6\" (UID: \"a5e8dab5-94da-46d0-8653-bcc21e849fc1\") " pod="openshift-console-operator/console-operator-7866497494-nj7h6" Jul 13 21:58:42.846556 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846434 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdh6n\" (UniqueName: \"kubernetes.io/projected/3a59df9a-555c-4676-9b35-d1f8b2752e82-kube-api-access-pdh6n\") pod \"insights-operator-68d9dd94cf-85gwn\" (UID: \"3a59df9a-555c-4676-9b35-d1f8b2752e82\") " pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:42.846556 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846467 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6cac02d-f5da-4569-894c-0328af666e63-image-registry-private-configuration\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.846556 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846490 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/779fd208-c240-4e2e-8eed-e76cd9a19023-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-stpjr\" (UID: \"779fd208-c240-4e2e-8eed-e76cd9a19023\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" Jul 13 21:58:42.846752 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846557 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.846752 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846586 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a59df9a-555c-4676-9b35-d1f8b2752e82-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-85gwn\" (UID: \"3a59df9a-555c-4676-9b35-d1f8b2752e82\") " pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:42.846752 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846611 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80fea354-ee7a-49af-8201-0bc32959c2d0-serving-cert\") pod \"service-ca-operator-7695fc6c8c-g4ds9\" (UID: \"80fea354-ee7a-49af-8201-0bc32959c2d0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g4ds9" Jul 13 21:58:42.846752 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846634 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-bound-sa-token\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.846752 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846656 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a5e8dab5-94da-46d0-8653-bcc21e849fc1-serving-cert\") pod \"console-operator-7866497494-nj7h6\" (UID: \"a5e8dab5-94da-46d0-8653-bcc21e849fc1\") " pod="openshift-console-operator/console-operator-7866497494-nj7h6" Jul 13 21:58:42.846752 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:42.846658 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 21:58:42.846752 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:42.846676 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f784dd4c9-l4qbc: secret "image-registry-tls" not found Jul 13 21:58:42.846752 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846697 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30ccc289-8e87-46a2-80b7-064bb0724db7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mscwk\" (UID: \"30ccc289-8e87-46a2-80b7-064bb0724db7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk" Jul 13 21:58:42.846752 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:42.846749 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls podName:d6cac02d-f5da-4569-894c-0328af666e63 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:43.346730805 +0000 UTC m=+36.003935570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls") pod "image-registry-7f784dd4c9-l4qbc" (UID: "d6cac02d-f5da-4569-894c-0328af666e63") : secret "image-registry-tls" not found Jul 13 21:58:42.847583 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846896 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a59df9a-555c-4676-9b35-d1f8b2752e82-serving-cert\") pod \"insights-operator-68d9dd94cf-85gwn\" (UID: \"3a59df9a-555c-4676-9b35-d1f8b2752e82\") " pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:42.847583 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846942 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3a59df9a-555c-4676-9b35-d1f8b2752e82-snapshots\") pod \"insights-operator-68d9dd94cf-85gwn\" (UID: \"3a59df9a-555c-4676-9b35-d1f8b2752e82\") " pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:42.847583 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.846981 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4c27\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-kube-api-access-b4c27\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.847583 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.847008 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5e8dab5-94da-46d0-8653-bcc21e849fc1-trusted-ca\") pod \"console-operator-7866497494-nj7h6\" (UID: \"a5e8dab5-94da-46d0-8653-bcc21e849fc1\") " pod="openshift-console-operator/console-operator-7866497494-nj7h6" Jul 13 21:58:42.847583 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.847031 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-stpjr\" (UID: \"779fd208-c240-4e2e-8eed-e76cd9a19023\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" Jul 13 21:58:42.847583 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.847059 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6cac02d-f5da-4569-894c-0328af666e63-installation-pull-secrets\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.847583 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.847067 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6cac02d-f5da-4569-894c-0328af666e63-registry-certificates\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.847583 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.847077 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80fea354-ee7a-49af-8201-0bc32959c2d0-config\") pod \"service-ca-operator-7695fc6c8c-g4ds9\" (UID: \"80fea354-ee7a-49af-8201-0bc32959c2d0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g4ds9" Jul 13 21:58:42.847583 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.847116 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6chz\" (UniqueName: \"kubernetes.io/projected/779fd208-c240-4e2e-8eed-e76cd9a19023-kube-api-access-j6chz\") pod \"cluster-monitoring-operator-5f7f494dd4-stpjr\" (UID: \"779fd208-c240-4e2e-8eed-e76cd9a19023\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" Jul 13 21:58:42.847583 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.847142 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a5e8dab5-94da-46d0-8653-bcc21e849fc1-config\") pod \"console-operator-7866497494-nj7h6\" (UID: \"a5e8dab5-94da-46d0-8653-bcc21e849fc1\") " pod="openshift-console-operator/console-operator-7866497494-nj7h6" Jul 13 21:58:42.847583 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.847151 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6cac02d-f5da-4569-894c-0328af666e63-ca-trust-extracted\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.847583 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.847206 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a59df9a-555c-4676-9b35-d1f8b2752e82-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-85gwn\" (UID: \"3a59df9a-555c-4676-9b35-d1f8b2752e82\") " pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:42.847583 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.847278 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6cac02d-f5da-4569-894c-0328af666e63-trusted-ca\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.848098 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.847420 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6cac02d-f5da-4569-894c-0328af666e63-ca-trust-extracted\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.848098 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.847739 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5e8dab5-94da-46d0-8653-bcc21e849fc1-trusted-ca\") pod \"console-operator-7866497494-nj7h6\" (UID: \"a5e8dab5-94da-46d0-8653-bcc21e849fc1\") " pod="openshift-console-operator/console-operator-7866497494-nj7h6" Jul 13 21:58:42.851068 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.851048 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6cac02d-f5da-4569-894c-0328af666e63-installation-pull-secrets\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.851068 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.851058 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a5e8dab5-94da-46d0-8653-bcc21e849fc1-serving-cert\") pod \"console-operator-7866497494-nj7h6\" (UID: \"a5e8dab5-94da-46d0-8653-bcc21e849fc1\") " pod="openshift-console-operator/console-operator-7866497494-nj7h6" Jul 13 21:58:42.851245 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.851225 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6cac02d-f5da-4569-894c-0328af666e63-image-registry-private-configuration\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.858733 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.858703 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndl6c\" (UniqueName: \"kubernetes.io/projected/a5e8dab5-94da-46d0-8653-bcc21e849fc1-kube-api-access-ndl6c\") pod \"console-operator-7866497494-nj7h6\" (UID: \"a5e8dab5-94da-46d0-8653-bcc21e849fc1\") " pod="openshift-console-operator/console-operator-7866497494-nj7h6" Jul 13 21:58:42.859488 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.859461 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-bound-sa-token\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.862231 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.862212 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpbf6\" (UniqueName: \"kubernetes.io/projected/2b79086d-97c9-4fe0-a960-a44518765dcf-kube-api-access-xpbf6\") pod \"volume-data-source-validator-78495597f6-sh5jg\" (UID: \"2b79086d-97c9-4fe0-a960-a44518765dcf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-sh5jg" Jul 13 21:58:42.862879 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.862864 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4c27\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-kube-api-access-b4c27\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:42.945963 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.945938 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-nj7h6" Jul 13 21:58:42.948180 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948161 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt944\" (UniqueName: \"kubernetes.io/projected/8d45c661-160c-4a4e-a7b8-c6e055315ae1-kube-api-access-rt944\") pod \"kube-storage-version-migrator-operator-799cb7f958-zghwn\" (UID: \"8d45c661-160c-4a4e-a7b8-c6e055315ae1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zghwn" Jul 13 21:58:42.948245 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948187 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fee610d-3565-4cfc-8d40-54ab4e09cac8-cert\") pod \"ingress-canary-thphz\" (UID: \"1fee610d-3565-4cfc-8d40-54ab4e09cac8\") " pod="openshift-ingress-canary/ingress-canary-thphz" Jul 13 21:58:42.948245 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948208 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a59df9a-555c-4676-9b35-d1f8b2752e82-tmp\") pod \"insights-operator-68d9dd94cf-85gwn\" (UID: \"3a59df9a-555c-4676-9b35-d1f8b2752e82\") " pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:42.948245 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948227 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:42.948345 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948251 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:42.948345 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948295 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmzk2\" (UniqueName: \"kubernetes.io/projected/80fea354-ee7a-49af-8201-0bc32959c2d0-kube-api-access-tmzk2\") pod \"service-ca-operator-7695fc6c8c-g4ds9\" (UID: \"80fea354-ee7a-49af-8201-0bc32959c2d0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g4ds9" Jul 13 21:58:42.948345 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948323 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs756\" (UniqueName: \"kubernetes.io/projected/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-kube-api-access-vs756\") pod \"dns-default-9dkjb\" (UID: \"f8a4145e-0e8f-43f6-83d7-ef573dbc5929\") " pod="openshift-dns/dns-default-9dkjb" Jul 13 21:58:42.948457 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948370 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdh6n\" (UniqueName: \"kubernetes.io/projected/3a59df9a-555c-4676-9b35-d1f8b2752e82-kube-api-access-pdh6n\") pod \"insights-operator-68d9dd94cf-85gwn\" (UID: \"3a59df9a-555c-4676-9b35-d1f8b2752e82\") " pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:42.948457 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948404 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d45c661-160c-4a4e-a7b8-c6e055315ae1-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-zghwn\" (UID: \"8d45c661-160c-4a4e-a7b8-c6e055315ae1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zghwn" Jul 13 21:58:42.948457 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948432 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m7lt\" (UniqueName: \"kubernetes.io/projected/1fee610d-3565-4cfc-8d40-54ab4e09cac8-kube-api-access-2m7lt\") pod \"ingress-canary-thphz\" (UID: \"1fee610d-3565-4cfc-8d40-54ab4e09cac8\") " pod="openshift-ingress-canary/ingress-canary-thphz" Jul 13 21:58:42.948618 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948461 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/779fd208-c240-4e2e-8eed-e76cd9a19023-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-stpjr\" (UID: \"779fd208-c240-4e2e-8eed-e76cd9a19023\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" Jul 13 21:58:42.948618 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948525 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a59df9a-555c-4676-9b35-d1f8b2752e82-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-85gwn\" (UID: \"3a59df9a-555c-4676-9b35-d1f8b2752e82\") " pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:42.948618 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948558 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a59df9a-555c-4676-9b35-d1f8b2752e82-tmp\") pod \"insights-operator-68d9dd94cf-85gwn\" (UID: \"3a59df9a-555c-4676-9b35-d1f8b2752e82\") " pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:42.948618 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948565 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80fea354-ee7a-49af-8201-0bc32959c2d0-serving-cert\") pod \"service-ca-operator-7695fc6c8c-g4ds9\" (UID: \"80fea354-ee7a-49af-8201-0bc32959c2d0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g4ds9" Jul 13 21:58:42.948618 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948592 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-default-certificate\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:42.948618 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948617 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-config-volume\") pod \"dns-default-9dkjb\" (UID: \"f8a4145e-0e8f-43f6-83d7-ef573dbc5929\") " pod="openshift-dns/dns-default-9dkjb" Jul 13 21:58:42.948825 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948648 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls\") pod \"dns-default-9dkjb\" (UID: \"f8a4145e-0e8f-43f6-83d7-ef573dbc5929\") " pod="openshift-dns/dns-default-9dkjb" Jul 13 21:58:42.948825 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948674 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30ccc289-8e87-46a2-80b7-064bb0724db7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mscwk\" (UID: \"30ccc289-8e87-46a2-80b7-064bb0724db7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk" Jul 13 21:58:42.948825 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948721 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a59df9a-555c-4676-9b35-d1f8b2752e82-serving-cert\") pod \"insights-operator-68d9dd94cf-85gwn\" (UID: \"3a59df9a-555c-4676-9b35-d1f8b2752e82\") " pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:42.948825 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:42.948749 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 21:58:42.948825 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948759 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3a59df9a-555c-4676-9b35-d1f8b2752e82-snapshots\") pod \"insights-operator-68d9dd94cf-85gwn\" (UID: \"3a59df9a-555c-4676-9b35-d1f8b2752e82\") " pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:42.948825 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948788 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d45c661-160c-4a4e-a7b8-c6e055315ae1-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-zghwn\" (UID: \"8d45c661-160c-4a4e-a7b8-c6e055315ae1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zghwn" Jul 13 21:58:42.948825 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:42.948816 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ccc289-8e87-46a2-80b7-064bb0724db7-samples-operator-tls podName:30ccc289-8e87-46a2-80b7-064bb0724db7 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:43.448796261 +0000 UTC m=+36.106001010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30ccc289-8e87-46a2-80b7-064bb0724db7-samples-operator-tls") pod "cluster-samples-operator-65db858894-mscwk" (UID: "30ccc289-8e87-46a2-80b7-064bb0724db7") : secret "samples-operator-tls" not found Jul 13 21:58:42.949165 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948874 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-stpjr\" (UID: \"779fd208-c240-4e2e-8eed-e76cd9a19023\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" Jul 13 21:58:42.949165 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948915 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80fea354-ee7a-49af-8201-0bc32959c2d0-config\") pod \"service-ca-operator-7695fc6c8c-g4ds9\" (UID: \"80fea354-ee7a-49af-8201-0bc32959c2d0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g4ds9" Jul 13 21:58:42.949165 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948942 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6chz\" (UniqueName: \"kubernetes.io/projected/779fd208-c240-4e2e-8eed-e76cd9a19023-kube-api-access-j6chz\") pod \"cluster-monitoring-operator-5f7f494dd4-stpjr\" (UID: \"779fd208-c240-4e2e-8eed-e76cd9a19023\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" Jul 13 21:58:42.949165 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.948975 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a59df9a-555c-4676-9b35-d1f8b2752e82-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-85gwn\" (UID: \"3a59df9a-555c-4676-9b35-d1f8b2752e82\") " pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:42.949165 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.949008 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kqv8\" (UniqueName: \"kubernetes.io/projected/d793005d-2131-4e53-8054-703b36d050b8-kube-api-access-8kqv8\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:42.949165 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.949050 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-stats-auth\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:42.949165 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.949077 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-tmp-dir\") pod \"dns-default-9dkjb\" (UID: \"f8a4145e-0e8f-43f6-83d7-ef573dbc5929\") " pod="openshift-dns/dns-default-9dkjb" Jul 13 21:58:42.949165 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.949106 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxlfm\" (UniqueName: \"kubernetes.io/projected/2c6dc74b-69a0-453a-94be-8f5006659d1e-kube-api-access-gxlfm\") pod \"network-check-source-869dcfcdcd-n55wp\" (UID: \"2c6dc74b-69a0-453a-94be-8f5006659d1e\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-n55wp" Jul 13 21:58:42.949165 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.949132 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfj2z\" (UniqueName: \"kubernetes.io/projected/30ccc289-8e87-46a2-80b7-064bb0724db7-kube-api-access-sfj2z\") pod \"cluster-samples-operator-65db858894-mscwk\" (UID: \"30ccc289-8e87-46a2-80b7-064bb0724db7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk" Jul 13 21:58:42.949625 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.949189 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/779fd208-c240-4e2e-8eed-e76cd9a19023-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-stpjr\" (UID: \"779fd208-c240-4e2e-8eed-e76cd9a19023\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" Jul 13 21:58:42.949625 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.949265 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3a59df9a-555c-4676-9b35-d1f8b2752e82-snapshots\") pod \"insights-operator-68d9dd94cf-85gwn\" (UID: \"3a59df9a-555c-4676-9b35-d1f8b2752e82\") " pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:42.949625 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:42.949365 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 21:58:42.949625 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:42.949412 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls podName:779fd208-c240-4e2e-8eed-e76cd9a19023 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:43.449396179 +0000 UTC m=+36.106600940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-stpjr" (UID: "779fd208-c240-4e2e-8eed-e76cd9a19023") : secret "cluster-monitoring-operator-tls" not found Jul 13 21:58:42.949625 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.949422 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a59df9a-555c-4676-9b35-d1f8b2752e82-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-85gwn\" (UID: \"3a59df9a-555c-4676-9b35-d1f8b2752e82\") " pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:42.949625 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.949584 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80fea354-ee7a-49af-8201-0bc32959c2d0-config\") pod \"service-ca-operator-7695fc6c8c-g4ds9\" (UID: \"80fea354-ee7a-49af-8201-0bc32959c2d0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g4ds9" Jul 13 21:58:42.949851 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.949829 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a59df9a-555c-4676-9b35-d1f8b2752e82-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-85gwn\" (UID: \"3a59df9a-555c-4676-9b35-d1f8b2752e82\") " pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:42.950624 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.950602 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80fea354-ee7a-49af-8201-0bc32959c2d0-serving-cert\") pod \"service-ca-operator-7695fc6c8c-g4ds9\" (UID: \"80fea354-ee7a-49af-8201-0bc32959c2d0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g4ds9" Jul 13 21:58:42.950971 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.950953 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a59df9a-555c-4676-9b35-d1f8b2752e82-serving-cert\") pod \"insights-operator-68d9dd94cf-85gwn\" (UID: \"3a59df9a-555c-4676-9b35-d1f8b2752e82\") " pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:42.955777 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.955762 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-sh5jg" Jul 13 21:58:42.958103 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.958084 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmzk2\" (UniqueName: \"kubernetes.io/projected/80fea354-ee7a-49af-8201-0bc32959c2d0-kube-api-access-tmzk2\") pod \"service-ca-operator-7695fc6c8c-g4ds9\" (UID: \"80fea354-ee7a-49af-8201-0bc32959c2d0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g4ds9" Jul 13 21:58:42.958434 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.958415 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfj2z\" (UniqueName: \"kubernetes.io/projected/30ccc289-8e87-46a2-80b7-064bb0724db7-kube-api-access-sfj2z\") pod \"cluster-samples-operator-65db858894-mscwk\" (UID: \"30ccc289-8e87-46a2-80b7-064bb0724db7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk" Jul 13 21:58:42.958513 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.958491 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdh6n\" (UniqueName: \"kubernetes.io/projected/3a59df9a-555c-4676-9b35-d1f8b2752e82-kube-api-access-pdh6n\") pod \"insights-operator-68d9dd94cf-85gwn\" (UID: \"3a59df9a-555c-4676-9b35-d1f8b2752e82\") " pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:42.959343 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.959326 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6chz\" (UniqueName: \"kubernetes.io/projected/779fd208-c240-4e2e-8eed-e76cd9a19023-kube-api-access-j6chz\") pod \"cluster-monitoring-operator-5f7f494dd4-stpjr\" (UID: \"779fd208-c240-4e2e-8eed-e76cd9a19023\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" Jul 13 21:58:42.959604 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:42.959587 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxlfm\" (UniqueName: \"kubernetes.io/projected/2c6dc74b-69a0-453a-94be-8f5006659d1e-kube-api-access-gxlfm\") pod \"network-check-source-869dcfcdcd-n55wp\" (UID: \"2c6dc74b-69a0-453a-94be-8f5006659d1e\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-n55wp" Jul 13 21:58:43.022701 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.022678 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g4ds9" Jul 13 21:58:43.050177 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.050150 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d45c661-160c-4a4e-a7b8-c6e055315ae1-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-zghwn\" (UID: \"8d45c661-160c-4a4e-a7b8-c6e055315ae1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zghwn" Jul 13 21:58:43.050300 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.050200 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kqv8\" (UniqueName: \"kubernetes.io/projected/d793005d-2131-4e53-8054-703b36d050b8-kube-api-access-8kqv8\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:43.050300 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.050221 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-stats-auth\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:43.050300 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.050241 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-tmp-dir\") pod \"dns-default-9dkjb\" (UID: \"f8a4145e-0e8f-43f6-83d7-ef573dbc5929\") " pod="openshift-dns/dns-default-9dkjb" Jul 13 21:58:43.050300 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.050261 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt944\" (UniqueName: \"kubernetes.io/projected/8d45c661-160c-4a4e-a7b8-c6e055315ae1-kube-api-access-rt944\") pod \"kube-storage-version-migrator-operator-799cb7f958-zghwn\" (UID: \"8d45c661-160c-4a4e-a7b8-c6e055315ae1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zghwn" Jul 13 21:58:43.050300 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.050276 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fee610d-3565-4cfc-8d40-54ab4e09cac8-cert\") pod \"ingress-canary-thphz\" (UID: \"1fee610d-3565-4cfc-8d40-54ab4e09cac8\") " pod="openshift-ingress-canary/ingress-canary-thphz" Jul 13 21:58:43.050300 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.050296 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:43.050472 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.050310 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:43.050472 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.050327 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs756\" (UniqueName: \"kubernetes.io/projected/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-kube-api-access-vs756\") pod \"dns-default-9dkjb\" (UID: \"f8a4145e-0e8f-43f6-83d7-ef573dbc5929\") " pod="openshift-dns/dns-default-9dkjb" Jul 13 21:58:43.050472 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.050347 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d45c661-160c-4a4e-a7b8-c6e055315ae1-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-zghwn\" (UID: \"8d45c661-160c-4a4e-a7b8-c6e055315ae1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zghwn" Jul 13 21:58:43.050472 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.050363 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m7lt\" (UniqueName: \"kubernetes.io/projected/1fee610d-3565-4cfc-8d40-54ab4e09cac8-kube-api-access-2m7lt\") pod \"ingress-canary-thphz\" (UID: \"1fee610d-3565-4cfc-8d40-54ab4e09cac8\") " pod="openshift-ingress-canary/ingress-canary-thphz" Jul 13 21:58:43.050472 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.050402 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-default-certificate\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:43.050472 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.050418 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-config-volume\") pod \"dns-default-9dkjb\" (UID: \"f8a4145e-0e8f-43f6-83d7-ef573dbc5929\") " pod="openshift-dns/dns-default-9dkjb" Jul 13 21:58:43.050472 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.050432 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls\") pod \"dns-default-9dkjb\" (UID: \"f8a4145e-0e8f-43f6-83d7-ef573dbc5929\") " pod="openshift-dns/dns-default-9dkjb" Jul 13 21:58:43.050796 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:43.050526 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 21:58:43.050796 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:43.050609 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls podName:f8a4145e-0e8f-43f6-83d7-ef573dbc5929 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:43.550592524 +0000 UTC m=+36.207797276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls") pod "dns-default-9dkjb" (UID: "f8a4145e-0e8f-43f6-83d7-ef573dbc5929") : secret "dns-default-metrics-tls" not found Jul 13 21:58:43.050796 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:43.050692 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle podName:d793005d-2131-4e53-8054-703b36d050b8 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:43.550671074 +0000 UTC m=+36.207875842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle") pod "router-default-768d869684-xh5gb" (UID: "d793005d-2131-4e53-8054-703b36d050b8") : configmap references non-existent config key: service-ca.crt Jul 13 21:58:43.050958 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.050852 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-tmp-dir\") pod \"dns-default-9dkjb\" (UID: \"f8a4145e-0e8f-43f6-83d7-ef573dbc5929\") " pod="openshift-dns/dns-default-9dkjb" Jul 13 21:58:43.051007 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:43.050989 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 21:58:43.051057 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:43.051031 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs podName:d793005d-2131-4e53-8054-703b36d050b8 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:43.551021011 +0000 UTC m=+36.208225760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs") pod "router-default-768d869684-xh5gb" (UID: "d793005d-2131-4e53-8054-703b36d050b8") : secret "router-metrics-certs-default" not found Jul 13 21:58:43.051594 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.051555 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d45c661-160c-4a4e-a7b8-c6e055315ae1-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-zghwn\" (UID: \"8d45c661-160c-4a4e-a7b8-c6e055315ae1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zghwn" Jul 13 21:58:43.051896 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.051867 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-config-volume\") pod \"dns-default-9dkjb\" (UID: \"f8a4145e-0e8f-43f6-83d7-ef573dbc5929\") " pod="openshift-dns/dns-default-9dkjb" Jul 13 21:58:43.052729 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.052684 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d45c661-160c-4a4e-a7b8-c6e055315ae1-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-zghwn\" (UID: \"8d45c661-160c-4a4e-a7b8-c6e055315ae1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zghwn" Jul 13 21:58:43.053160 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.053129 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-stats-auth\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:43.053458 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.053436 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fee610d-3565-4cfc-8d40-54ab4e09cac8-cert\") pod \"ingress-canary-thphz\" (UID: \"1fee610d-3565-4cfc-8d40-54ab4e09cac8\") " pod="openshift-ingress-canary/ingress-canary-thphz" Jul 13 21:58:43.053684 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.053668 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-default-certificate\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:43.060170 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.059774 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m7lt\" (UniqueName: \"kubernetes.io/projected/1fee610d-3565-4cfc-8d40-54ab4e09cac8-kube-api-access-2m7lt\") pod \"ingress-canary-thphz\" (UID: \"1fee610d-3565-4cfc-8d40-54ab4e09cac8\") " pod="openshift-ingress-canary/ingress-canary-thphz" Jul 13 21:58:43.060170 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.059873 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" Jul 13 21:58:43.060170 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.060151 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt944\" (UniqueName: \"kubernetes.io/projected/8d45c661-160c-4a4e-a7b8-c6e055315ae1-kube-api-access-rt944\") pod \"kube-storage-version-migrator-operator-799cb7f958-zghwn\" (UID: \"8d45c661-160c-4a4e-a7b8-c6e055315ae1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zghwn" Jul 13 21:58:43.060362 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.060257 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kqv8\" (UniqueName: \"kubernetes.io/projected/d793005d-2131-4e53-8054-703b36d050b8-kube-api-access-8kqv8\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:43.060603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.060586 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs756\" (UniqueName: \"kubernetes.io/projected/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-kube-api-access-vs756\") pod \"dns-default-9dkjb\" (UID: \"f8a4145e-0e8f-43f6-83d7-ef573dbc5929\") " pod="openshift-dns/dns-default-9dkjb" Jul 13 21:58:43.072682 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.072594 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-n55wp" Jul 13 21:58:43.099128 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.098680 2569 generic.go:356] "Generic (PLEG): container finished" podID="83271b6e-2653-44d9-8ddb-03bed0f66e39" containerID="70db8049768c50e301cca4740a36a3d0968607f7beb835a75ff1ed9eeee50bc5" exitCode=0 Jul 13 21:58:43.099128 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.098761 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwskc" event={"ID":"83271b6e-2653-44d9-8ddb-03bed0f66e39","Type":"ContainerDied","Data":"70db8049768c50e301cca4740a36a3d0968607f7beb835a75ff1ed9eeee50bc5"} Jul 13 21:58:43.099128 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.099004 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zghwn" Jul 13 21:58:43.129308 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.128190 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-thphz" Jul 13 21:58:43.198861 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.198692 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-nj7h6"] Jul 13 21:58:43.220416 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.217330 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-sh5jg"] Jul 13 21:58:43.242247 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:43.242184 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b79086d_97c9_4fe0_a960_a44518765dcf.slice/crio-c7dc001f772f9a1656dab8a6f6fb74caf2d80b110e35a5bbb66a245fd55ea14d WatchSource:0}: Error finding container c7dc001f772f9a1656dab8a6f6fb74caf2d80b110e35a5bbb66a245fd55ea14d: Status 404 returned error can't find the container with id c7dc001f772f9a1656dab8a6f6fb74caf2d80b110e35a5bbb66a245fd55ea14d Jul 13 21:58:43.277628 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.275599 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g4ds9"] Jul 13 21:58:43.314576 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.314547 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-85gwn"] Jul 13 21:58:43.317814 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:43.317782 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a59df9a_555c_4676_9b35_d1f8b2752e82.slice/crio-cdfd50f0755d2a7478ef19f7211d3c941e01c909f8262875a43ac89ec1a3294b WatchSource:0}: Error finding container cdfd50f0755d2a7478ef19f7211d3c941e01c909f8262875a43ac89ec1a3294b: Status 404 returned error can't find the container with id cdfd50f0755d2a7478ef19f7211d3c941e01c909f8262875a43ac89ec1a3294b Jul 13 21:58:43.330771 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.330749 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-n55wp"] Jul 13 21:58:43.333789 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:43.333763 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6dc74b_69a0_453a_94be_8f5006659d1e.slice/crio-8e1ca3b7609d082ef1106dbd19c3cbffe13eb2b10abf44860b55847f944e301a WatchSource:0}: Error finding container 8e1ca3b7609d082ef1106dbd19c3cbffe13eb2b10abf44860b55847f944e301a: Status 404 returned error can't find the container with id 8e1ca3b7609d082ef1106dbd19c3cbffe13eb2b10abf44860b55847f944e301a Jul 13 21:58:43.338684 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.338663 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zghwn"] Jul 13 21:58:43.355094 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.355074 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-thphz"] Jul 13 21:58:43.357181 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.357162 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:43.357337 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:43.357321 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 21:58:43.357381 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:43.357339 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f784dd4c9-l4qbc: secret "image-registry-tls" not found Jul 13 21:58:43.357777 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:43.357767 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls podName:d6cac02d-f5da-4569-894c-0328af666e63 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:44.357750342 +0000 UTC m=+37.014955090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls") pod "image-registry-7f784dd4c9-l4qbc" (UID: "d6cac02d-f5da-4569-894c-0328af666e63") : secret "image-registry-tls" not found Jul 13 21:58:43.363888 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:43.363855 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d45c661_160c_4a4e_a7b8_c6e055315ae1.slice/crio-3c6ccec485be8169bcbbfba148f96aa62616ba30cd8d9ff26fcf14e2bc587b2f WatchSource:0}: Error finding container 3c6ccec485be8169bcbbfba148f96aa62616ba30cd8d9ff26fcf14e2bc587b2f: Status 404 returned error can't find the container with id 3c6ccec485be8169bcbbfba148f96aa62616ba30cd8d9ff26fcf14e2bc587b2f Jul 13 21:58:43.364868 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:43.364844 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fee610d_3565_4cfc_8d40_54ab4e09cac8.slice/crio-4cd4b7f5a05c8a938dd432a1294bb708a5640dd5eb502501eeb25a422ba48d7a WatchSource:0}: Error finding container 4cd4b7f5a05c8a938dd432a1294bb708a5640dd5eb502501eeb25a422ba48d7a: Status 404 returned error can't find the container with id 4cd4b7f5a05c8a938dd432a1294bb708a5640dd5eb502501eeb25a422ba48d7a Jul 13 21:58:43.458161 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.458129 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30ccc289-8e87-46a2-80b7-064bb0724db7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mscwk\" (UID: \"30ccc289-8e87-46a2-80b7-064bb0724db7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk" Jul 13 21:58:43.458258 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.458193 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-stpjr\" (UID: \"779fd208-c240-4e2e-8eed-e76cd9a19023\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" Jul 13 21:58:43.458299 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:43.458277 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 21:58:43.458338 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:43.458306 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 21:58:43.458338 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:43.458336 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ccc289-8e87-46a2-80b7-064bb0724db7-samples-operator-tls podName:30ccc289-8e87-46a2-80b7-064bb0724db7 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:44.458319953 +0000 UTC m=+37.115524700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30ccc289-8e87-46a2-80b7-064bb0724db7-samples-operator-tls") pod "cluster-samples-operator-65db858894-mscwk" (UID: "30ccc289-8e87-46a2-80b7-064bb0724db7") : secret "samples-operator-tls" not found Jul 13 21:58:43.458407 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:43.458350 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls podName:779fd208-c240-4e2e-8eed-e76cd9a19023 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:44.458343394 +0000 UTC m=+37.115548140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-stpjr" (UID: "779fd208-c240-4e2e-8eed-e76cd9a19023") : secret "cluster-monitoring-operator-tls" not found Jul 13 21:58:43.558518 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.558462 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:43.558518 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.558493 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:43.558651 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.558547 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls\") pod \"dns-default-9dkjb\" (UID: \"f8a4145e-0e8f-43f6-83d7-ef573dbc5929\") " pod="openshift-dns/dns-default-9dkjb" Jul 13 21:58:43.558651 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:43.558621 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 21:58:43.558651 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:43.558643 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle podName:d793005d-2131-4e53-8054-703b36d050b8 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:44.558627835 +0000 UTC m=+37.215832598 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle") pod "router-default-768d869684-xh5gb" (UID: "d793005d-2131-4e53-8054-703b36d050b8") : configmap references non-existent config key: service-ca.crt Jul 13 21:58:43.558792 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:43.558663 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs podName:d793005d-2131-4e53-8054-703b36d050b8 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:44.558656695 +0000 UTC m=+37.215861441 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs") pod "router-default-768d869684-xh5gb" (UID: "d793005d-2131-4e53-8054-703b36d050b8") : secret "router-metrics-certs-default" not found Jul 13 21:58:43.558792 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:43.558647 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 21:58:43.558792 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:43.558759 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls podName:f8a4145e-0e8f-43f6-83d7-ef573dbc5929 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:44.558739964 +0000 UTC m=+37.215944724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls") pod "dns-default-9dkjb" (UID: "f8a4145e-0e8f-43f6-83d7-ef573dbc5929") : secret "dns-default-metrics-tls" not found Jul 13 21:58:43.925608 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.925120 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:58:43.925608 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.925365 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:58:43.925608 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.925595 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:58:43.929742 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.929067 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vvmxw\"" Jul 13 21:58:43.929742 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.929315 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 21:58:43.929742 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.929586 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 21:58:43.930370 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:43.930200 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-djqhc\"" Jul 13 21:58:44.107471 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:44.107425 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-thphz" event={"ID":"1fee610d-3565-4cfc-8d40-54ab4e09cac8","Type":"ContainerStarted","Data":"4cd4b7f5a05c8a938dd432a1294bb708a5640dd5eb502501eeb25a422ba48d7a"} Jul 13 21:58:44.111666 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:44.111628 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zghwn" event={"ID":"8d45c661-160c-4a4e-a7b8-c6e055315ae1","Type":"ContainerStarted","Data":"3c6ccec485be8169bcbbfba148f96aa62616ba30cd8d9ff26fcf14e2bc587b2f"} Jul 13 21:58:44.117938 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:44.117852 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g4ds9" event={"ID":"80fea354-ee7a-49af-8201-0bc32959c2d0","Type":"ContainerStarted","Data":"eb79c11395750bb06b59334edbae68e89bf791f7c805042095cccd070fbe4ef7"} Jul 13 21:58:44.119936 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:44.119874 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-nj7h6" event={"ID":"a5e8dab5-94da-46d0-8653-bcc21e849fc1","Type":"ContainerStarted","Data":"371b63ae0a60f77de273ec0741ebd266d8fc950bd9e3b7c5aab34bcfedf5c025"} Jul 13 21:58:44.125886 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:44.125825 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" event={"ID":"3a59df9a-555c-4676-9b35-d1f8b2752e82","Type":"ContainerStarted","Data":"cdfd50f0755d2a7478ef19f7211d3c941e01c909f8262875a43ac89ec1a3294b"} Jul 13 21:58:44.128599 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:44.128556 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-sh5jg" event={"ID":"2b79086d-97c9-4fe0-a960-a44518765dcf","Type":"ContainerStarted","Data":"c7dc001f772f9a1656dab8a6f6fb74caf2d80b110e35a5bbb66a245fd55ea14d"} Jul 13 21:58:44.142355 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:44.140340 2569 generic.go:356] "Generic (PLEG): container finished" podID="83271b6e-2653-44d9-8ddb-03bed0f66e39" containerID="c16d4a9fc67bc3f6ed8369e8e1998960f1f79fbb773015fc4799442e324b264b" exitCode=0 Jul 13 21:58:44.142355 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:44.140412 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwskc" event={"ID":"83271b6e-2653-44d9-8ddb-03bed0f66e39","Type":"ContainerDied","Data":"c16d4a9fc67bc3f6ed8369e8e1998960f1f79fbb773015fc4799442e324b264b"} Jul 13 21:58:44.146662 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:44.144955 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-n55wp" event={"ID":"2c6dc74b-69a0-453a-94be-8f5006659d1e","Type":"ContainerStarted","Data":"8e1ca3b7609d082ef1106dbd19c3cbffe13eb2b10abf44860b55847f944e301a"} Jul 13 21:58:44.366059 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:44.366029 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:44.366259 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:44.366217 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 21:58:44.366259 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:44.366241 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f784dd4c9-l4qbc: secret "image-registry-tls" not found Jul 13 21:58:44.366388 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:44.366296 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls podName:d6cac02d-f5da-4569-894c-0328af666e63 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:46.366276827 +0000 UTC m=+39.023481590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls") pod "image-registry-7f784dd4c9-l4qbc" (UID: "d6cac02d-f5da-4569-894c-0328af666e63") : secret "image-registry-tls" not found Jul 13 21:58:44.467720 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:44.467640 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-stpjr\" (UID: \"779fd208-c240-4e2e-8eed-e76cd9a19023\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" Jul 13 21:58:44.467902 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:44.467793 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30ccc289-8e87-46a2-80b7-064bb0724db7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mscwk\" (UID: \"30ccc289-8e87-46a2-80b7-064bb0724db7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk" Jul 13 21:58:44.467973 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:44.467916 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 21:58:44.468023 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:44.467982 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ccc289-8e87-46a2-80b7-064bb0724db7-samples-operator-tls podName:30ccc289-8e87-46a2-80b7-064bb0724db7 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:46.467962156 +0000 UTC m=+39.125166906 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30ccc289-8e87-46a2-80b7-064bb0724db7-samples-operator-tls") pod "cluster-samples-operator-65db858894-mscwk" (UID: "30ccc289-8e87-46a2-80b7-064bb0724db7") : secret "samples-operator-tls" not found Jul 13 21:58:44.468256 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:44.468176 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 21:58:44.468256 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:44.468225 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls podName:779fd208-c240-4e2e-8eed-e76cd9a19023 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:46.468212276 +0000 UTC m=+39.125417038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-stpjr" (UID: "779fd208-c240-4e2e-8eed-e76cd9a19023") : secret "cluster-monitoring-operator-tls" not found Jul 13 21:58:44.568735 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:44.568622 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:44.568735 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:44.568672 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:44.569026 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:44.568749 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls\") pod \"dns-default-9dkjb\" (UID: \"f8a4145e-0e8f-43f6-83d7-ef573dbc5929\") " pod="openshift-dns/dns-default-9dkjb" Jul 13 21:58:44.569026 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:44.568938 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 21:58:44.569026 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:44.569001 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls podName:f8a4145e-0e8f-43f6-83d7-ef573dbc5929 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:46.568981392 +0000 UTC m=+39.226186154 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls") pod "dns-default-9dkjb" (UID: "f8a4145e-0e8f-43f6-83d7-ef573dbc5929") : secret "dns-default-metrics-tls" not found Jul 13 21:58:44.569182 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:44.569081 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle podName:d793005d-2131-4e53-8054-703b36d050b8 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:46.569070203 +0000 UTC m=+39.226274953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle") pod "router-default-768d869684-xh5gb" (UID: "d793005d-2131-4e53-8054-703b36d050b8") : configmap references non-existent config key: service-ca.crt Jul 13 21:58:44.569182 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:44.569140 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 21:58:44.569182 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:44.569171 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs podName:d793005d-2131-4e53-8054-703b36d050b8 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:46.569160755 +0000 UTC m=+39.226365503 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs") pod "router-default-768d869684-xh5gb" (UID: "d793005d-2131-4e53-8054-703b36d050b8") : secret "router-metrics-certs-default" not found Jul 13 21:58:45.156104 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:45.156066 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-sh5jg" event={"ID":"2b79086d-97c9-4fe0-a960-a44518765dcf","Type":"ContainerStarted","Data":"2b9227673f0952592de781426ec1e5e79f1057897fa1adbd60389509e28672c9"} Jul 13 21:58:45.168895 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:45.168227 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwskc" event={"ID":"83271b6e-2653-44d9-8ddb-03bed0f66e39","Type":"ContainerStarted","Data":"6f5ed08e1227c58d23619c910ddf146171d2f4dc3d3c5ffc3adf75ee3955cfb2"} Jul 13 21:58:45.173096 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:45.171636 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-sh5jg" podStartSLOduration=22.457658677 podStartE2EDuration="24.171618741s" totalImagesPullingTime="1.713960064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:45.170673475 +0000 UTC m=+37.827878244" watchObservedRunningTime="2026-07-13 21:58:45.171618741 +0000 UTC m=+37.828823510" Jul 13 21:58:45.194824 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:45.193741 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dwskc" podStartSLOduration=7.338555816 podStartE2EDuration="37.193721725s" totalImagesPullingTime="29.855165909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:58:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:45.19229008 +0000 UTC m=+37.849494849" watchObservedRunningTime="2026-07-13 21:58:45.193721725 +0000 UTC m=+37.850926494" Jul 13 21:58:45.338262 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:45.338149 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vpg2f_3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0/konnectivity-agent/0.log" limit=1024 Jul 13 21:58:45.421859 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:45.421822 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vpg2f_3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0/konnectivity-agent/0.log" limit=1024 Jul 13 21:58:46.390081 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:46.390045 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:46.390461 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:46.390184 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 21:58:46.390461 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:46.390204 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f784dd4c9-l4qbc: secret "image-registry-tls" not found Jul 13 21:58:46.390461 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:46.390271 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls podName:d6cac02d-f5da-4569-894c-0328af666e63 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:50.39025267 +0000 UTC m=+43.047457423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls") pod "image-registry-7f784dd4c9-l4qbc" (UID: "d6cac02d-f5da-4569-894c-0328af666e63") : secret "image-registry-tls" not found Jul 13 21:58:46.490957 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:46.490753 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30ccc289-8e87-46a2-80b7-064bb0724db7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mscwk\" (UID: \"30ccc289-8e87-46a2-80b7-064bb0724db7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk" Jul 13 21:58:46.491136 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:46.491004 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-stpjr\" (UID: \"779fd208-c240-4e2e-8eed-e76cd9a19023\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" Jul 13 21:58:46.491136 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:46.490918 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 21:58:46.491136 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:46.491103 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ccc289-8e87-46a2-80b7-064bb0724db7-samples-operator-tls podName:30ccc289-8e87-46a2-80b7-064bb0724db7 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:50.491072929 +0000 UTC m=+43.148277674 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30ccc289-8e87-46a2-80b7-064bb0724db7-samples-operator-tls") pod "cluster-samples-operator-65db858894-mscwk" (UID: "30ccc289-8e87-46a2-80b7-064bb0724db7") : secret "samples-operator-tls" not found Jul 13 21:58:46.491303 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:46.491138 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 21:58:46.491303 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:46.491182 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls podName:779fd208-c240-4e2e-8eed-e76cd9a19023 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:50.49116784 +0000 UTC m=+43.148372592 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-stpjr" (UID: "779fd208-c240-4e2e-8eed-e76cd9a19023") : secret "cluster-monitoring-operator-tls" not found Jul 13 21:58:46.591822 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:46.591786 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls\") pod \"dns-default-9dkjb\" (UID: \"f8a4145e-0e8f-43f6-83d7-ef573dbc5929\") " pod="openshift-dns/dns-default-9dkjb" Jul 13 21:58:46.591979 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:46.591912 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 21:58:46.591979 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:46.591927 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:46.591979 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:46.591971 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls podName:f8a4145e-0e8f-43f6-83d7-ef573dbc5929 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:50.591953101 +0000 UTC m=+43.249157859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls") pod "dns-default-9dkjb" (UID: "f8a4145e-0e8f-43f6-83d7-ef573dbc5929") : secret "dns-default-metrics-tls" not found Jul 13 21:58:46.591979 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:46.591958 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:46.592151 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:46.592090 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle podName:d793005d-2131-4e53-8054-703b36d050b8 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:50.592072779 +0000 UTC m=+43.249277530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle") pod "router-default-768d869684-xh5gb" (UID: "d793005d-2131-4e53-8054-703b36d050b8") : configmap references non-existent config key: service-ca.crt Jul 13 21:58:46.592151 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:46.592096 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 21:58:46.592151 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:46.592139 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs podName:d793005d-2131-4e53-8054-703b36d050b8 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:50.592127374 +0000 UTC m=+43.249332125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs") pod "router-default-768d869684-xh5gb" (UID: "d793005d-2131-4e53-8054-703b36d050b8") : secret "router-metrics-certs-default" not found Jul 13 21:58:47.137596 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:47.137531 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vpg2f_3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0/konnectivity-agent/0.log" limit=1024 Jul 13 21:58:47.237515 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:47.237482 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vpg2f_3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0/konnectivity-agent/0.log" limit=1024 Jul 13 21:58:47.704708 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:47.704663 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vpg2f_3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0/konnectivity-agent/0.log" limit=1024 Jul 13 21:58:47.804565 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:47.804520 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vpg2f_3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0/konnectivity-agent/0.log" limit=1024 Jul 13 21:58:50.425424 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:50.425386 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:50.425818 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:50.425558 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 21:58:50.425818 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:50.425578 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f784dd4c9-l4qbc: secret "image-registry-tls" not found Jul 13 21:58:50.425818 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:50.425640 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls podName:d6cac02d-f5da-4569-894c-0328af666e63 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:58.425625154 +0000 UTC m=+51.082829900 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls") pod "image-registry-7f784dd4c9-l4qbc" (UID: "d6cac02d-f5da-4569-894c-0328af666e63") : secret "image-registry-tls" not found Jul 13 21:58:50.526364 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:50.526332 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30ccc289-8e87-46a2-80b7-064bb0724db7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mscwk\" (UID: \"30ccc289-8e87-46a2-80b7-064bb0724db7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk" Jul 13 21:58:50.526472 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:50.526386 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-stpjr\" (UID: \"779fd208-c240-4e2e-8eed-e76cd9a19023\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" Jul 13 21:58:50.526507 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:50.526477 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 21:58:50.526507 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:50.526483 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 21:58:50.526600 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:50.526559 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls podName:779fd208-c240-4e2e-8eed-e76cd9a19023 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:58.52652327 +0000 UTC m=+51.183728034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-stpjr" (UID: "779fd208-c240-4e2e-8eed-e76cd9a19023") : secret "cluster-monitoring-operator-tls" not found Jul 13 21:58:50.526600 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:50.526578 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ccc289-8e87-46a2-80b7-064bb0724db7-samples-operator-tls podName:30ccc289-8e87-46a2-80b7-064bb0724db7 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:58.526568163 +0000 UTC m=+51.183772923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30ccc289-8e87-46a2-80b7-064bb0724db7-samples-operator-tls") pod "cluster-samples-operator-65db858894-mscwk" (UID: "30ccc289-8e87-46a2-80b7-064bb0724db7") : secret "samples-operator-tls" not found Jul 13 21:58:50.627212 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:50.627189 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:50.627291 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:50.627220 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:50.627291 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:50.627281 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls\") pod \"dns-default-9dkjb\" (UID: \"f8a4145e-0e8f-43f6-83d7-ef573dbc5929\") " pod="openshift-dns/dns-default-9dkjb" Jul 13 21:58:50.627361 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:50.627339 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle podName:d793005d-2131-4e53-8054-703b36d050b8 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:58.627323221 +0000 UTC m=+51.284527978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle") pod "router-default-768d869684-xh5gb" (UID: "d793005d-2131-4e53-8054-703b36d050b8") : configmap references non-existent config key: service-ca.crt Jul 13 21:58:50.627410 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:50.627390 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 21:58:50.627461 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:50.627451 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs podName:d793005d-2131-4e53-8054-703b36d050b8 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:58.62743482 +0000 UTC m=+51.284639581 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs") pod "router-default-768d869684-xh5gb" (UID: "d793005d-2131-4e53-8054-703b36d050b8") : secret "router-metrics-certs-default" not found Jul 13 21:58:50.627507 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:50.627464 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 21:58:50.627507 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:50.627495 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls podName:f8a4145e-0e8f-43f6-83d7-ef573dbc5929 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:58.62748693 +0000 UTC m=+51.284691679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls") pod "dns-default-9dkjb" (UID: "f8a4145e-0e8f-43f6-83d7-ef573dbc5929") : secret "dns-default-metrics-tls" not found Jul 13 21:58:51.186065 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:51.186037 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/0.log" Jul 13 21:58:51.186226 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:51.186076 2569 generic.go:356] "Generic (PLEG): container finished" podID="a5e8dab5-94da-46d0-8653-bcc21e849fc1" containerID="1cdf9a6168b91864c7857fefdda4639b2ef84d4795d83367a622c232d6964cb4" exitCode=255 Jul 13 21:58:51.186226 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:51.186164 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-nj7h6" event={"ID":"a5e8dab5-94da-46d0-8653-bcc21e849fc1","Type":"ContainerDied","Data":"1cdf9a6168b91864c7857fefdda4639b2ef84d4795d83367a622c232d6964cb4"} Jul 13 21:58:51.186579 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:51.186371 2569 scope.go:117] "RemoveContainer" containerID="1cdf9a6168b91864c7857fefdda4639b2ef84d4795d83367a622c232d6964cb4" Jul 13 21:58:51.187878 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:51.187851 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" event={"ID":"3a59df9a-555c-4676-9b35-d1f8b2752e82","Type":"ContainerStarted","Data":"3dc61d28ccc0e5470607fecd448f21e97352685c123baa876b95bdca1f37582c"} Jul 13 21:58:51.192220 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:51.192187 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-n55wp" event={"ID":"2c6dc74b-69a0-453a-94be-8f5006659d1e","Type":"ContainerStarted","Data":"c2c2036c9a229e819d291460d35503ee71324133c5e570e4da3b172730ce81b5"} Jul 13 21:58:51.197224 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:51.197199 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-thphz" event={"ID":"1fee610d-3565-4cfc-8d40-54ab4e09cac8","Type":"ContainerStarted","Data":"8bfffd14fa05e88721ee97923e46906da329d06e5fe273b190a04bc012a5fb99"} Jul 13 21:58:51.212086 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:51.206665 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zghwn" event={"ID":"8d45c661-160c-4a4e-a7b8-c6e055315ae1","Type":"ContainerStarted","Data":"0d321fefdc72203948692ab7f00a4ad983821df9d72b7a070715ce5e72639b80"} Jul 13 21:58:51.212086 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:51.209371 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g4ds9" event={"ID":"80fea354-ee7a-49af-8201-0bc32959c2d0","Type":"ContainerStarted","Data":"6fcbe817cd703595c64cc2c36c818c7b5758e506203fbe018fdde07d311c8e24"} Jul 13 21:58:51.220936 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:51.220777 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-n55wp" podStartSLOduration=19.7122017 podStartE2EDuration="27.220764874s" totalImagesPullingTime="7.508563174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:51.220302554 +0000 UTC m=+43.877507323" watchObservedRunningTime="2026-07-13 21:58:51.220764874 +0000 UTC m=+43.877969619" Jul 13 21:58:51.238159 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:51.237646 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" podStartSLOduration=22.723993926 podStartE2EDuration="30.237631477s" totalImagesPullingTime="7.513637551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:51.236940785 +0000 UTC m=+43.894145554" watchObservedRunningTime="2026-07-13 21:58:51.237631477 +0000 UTC m=+43.894836248" Jul 13 21:58:51.255832 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:51.255789 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-thphz" podStartSLOduration=1.791603075 podStartE2EDuration="9.25575857s" totalImagesPullingTime="7.464155495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:51.25389186 +0000 UTC m=+43.911096629" watchObservedRunningTime="2026-07-13 21:58:51.25575857 +0000 UTC m=+43.912963338" Jul 13 21:58:51.272769 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:51.272709 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zghwn" podStartSLOduration=22.80481031 podStartE2EDuration="30.272692096s" totalImagesPullingTime="7.467881786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:51.270986792 +0000 UTC m=+43.928191564" watchObservedRunningTime="2026-07-13 21:58:51.272692096 +0000 UTC m=+43.929896871" Jul 13 21:58:52.215463 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.215434 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 21:58:52.216425 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.216400 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/0.log" Jul 13 21:58:52.216581 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.216451 2569 generic.go:356] "Generic (PLEG): container finished" podID="a5e8dab5-94da-46d0-8653-bcc21e849fc1" containerID="1f40c7fed419b8ef9c32459c9eeeec0dcf442dba88492d186835a37091160dca" exitCode=255 Jul 13 21:58:52.217158 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.217132 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-nj7h6" event={"ID":"a5e8dab5-94da-46d0-8653-bcc21e849fc1","Type":"ContainerDied","Data":"1f40c7fed419b8ef9c32459c9eeeec0dcf442dba88492d186835a37091160dca"} Jul 13 21:58:52.217322 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.217305 2569 scope.go:117] "RemoveContainer" containerID="1cdf9a6168b91864c7857fefdda4639b2ef84d4795d83367a622c232d6964cb4" Jul 13 21:58:52.217496 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.217474 2569 scope.go:117] "RemoveContainer" containerID="1f40c7fed419b8ef9c32459c9eeeec0dcf442dba88492d186835a37091160dca" Jul 13 21:58:52.217785 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:52.217720 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-nj7h6_openshift-console-operator(a5e8dab5-94da-46d0-8653-bcc21e849fc1)\"" pod="openshift-console-operator/console-operator-7866497494-nj7h6" podUID="a5e8dab5-94da-46d0-8653-bcc21e849fc1" Jul 13 21:58:52.235416 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.235376 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g4ds9" podStartSLOduration=23.690575049 podStartE2EDuration="31.235360764s" totalImagesPullingTime="7.544785715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:51.289784953 +0000 UTC m=+43.946989727" watchObservedRunningTime="2026-07-13 21:58:52.235360764 +0000 UTC m=+44.892565558" Jul 13 21:58:52.236285 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.236264 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bfh2f"] Jul 13 21:58:52.257415 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.257385 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vpg2f_3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0/konnectivity-agent/0.log" limit=1024 Jul 13 21:58:52.262110 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.262090 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bfh2f"] Jul 13 21:58:52.262237 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.262218 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:52.265204 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.264972 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 13 21:58:52.265512 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.265492 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 13 21:58:52.265701 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.265683 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tlwrc\"" Jul 13 21:58:52.328067 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.328028 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vpg2f_3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0/konnectivity-agent/0.log" limit=1024 Jul 13 21:58:52.342396 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.342360 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abaa946e-e127-4ff9-8072-23c540c62eb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:52.342544 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.342408 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtkm8\" (UniqueName: \"kubernetes.io/projected/abaa946e-e127-4ff9-8072-23c540c62eb5-kube-api-access-vtkm8\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:52.342544 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.342467 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abaa946e-e127-4ff9-8072-23c540c62eb5-crio-socket\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:52.342544 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.342489 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abaa946e-e127-4ff9-8072-23c540c62eb5-data-volume\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:52.342656 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.342555 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abaa946e-e127-4ff9-8072-23c540c62eb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:52.443022 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.442992 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtkm8\" (UniqueName: \"kubernetes.io/projected/abaa946e-e127-4ff9-8072-23c540c62eb5-kube-api-access-vtkm8\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:52.443171 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.443052 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abaa946e-e127-4ff9-8072-23c540c62eb5-crio-socket\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:52.443171 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.443086 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abaa946e-e127-4ff9-8072-23c540c62eb5-data-volume\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:52.443171 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.443123 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abaa946e-e127-4ff9-8072-23c540c62eb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:52.443327 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:52.443209 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 21:58:52.443327 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:52.443263 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/abaa946e-e127-4ff9-8072-23c540c62eb5-kube-rbac-proxy-cm podName:abaa946e-e127-4ff9-8072-23c540c62eb5 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:52.943245946 +0000 UTC m=+45.600450707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/abaa946e-e127-4ff9-8072-23c540c62eb5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bfh2f" (UID: "abaa946e-e127-4ff9-8072-23c540c62eb5") : configmap "kube-rbac-proxy" not found Jul 13 21:58:52.443327 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.443256 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abaa946e-e127-4ff9-8072-23c540c62eb5-crio-socket\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:52.443452 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.443381 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abaa946e-e127-4ff9-8072-23c540c62eb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:52.443498 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:52.443469 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 21:58:52.443498 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.443472 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abaa946e-e127-4ff9-8072-23c540c62eb5-data-volume\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:52.443595 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:52.443506 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abaa946e-e127-4ff9-8072-23c540c62eb5-insights-runtime-extractor-tls podName:abaa946e-e127-4ff9-8072-23c540c62eb5 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:52.943494908 +0000 UTC m=+45.600699656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abaa946e-e127-4ff9-8072-23c540c62eb5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bfh2f" (UID: "abaa946e-e127-4ff9-8072-23c540c62eb5") : secret "insights-runtime-extractor-tls" not found Jul 13 21:58:52.453282 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.453259 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtkm8\" (UniqueName: \"kubernetes.io/projected/abaa946e-e127-4ff9-8072-23c540c62eb5-kube-api-access-vtkm8\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:52.916951 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.916919 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vpg2f_3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0/konnectivity-agent/0.log" limit=1024 Jul 13 21:58:52.941705 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.941673 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rnlsb"] Jul 13 21:58:52.946831 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.946804 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abaa946e-e127-4ff9-8072-23c540c62eb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:52.946935 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.946871 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abaa946e-e127-4ff9-8072-23c540c62eb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:52.946998 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:52.946933 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 21:58:52.946998 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:52.946948 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 21:58:52.946998 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:52.946993 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abaa946e-e127-4ff9-8072-23c540c62eb5-insights-runtime-extractor-tls podName:abaa946e-e127-4ff9-8072-23c540c62eb5 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:53.946978896 +0000 UTC m=+46.604183642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abaa946e-e127-4ff9-8072-23c540c62eb5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bfh2f" (UID: "abaa946e-e127-4ff9-8072-23c540c62eb5") : secret "insights-runtime-extractor-tls" not found Jul 13 21:58:52.947142 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:52.947007 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/abaa946e-e127-4ff9-8072-23c540c62eb5-kube-rbac-proxy-cm podName:abaa946e-e127-4ff9-8072-23c540c62eb5 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:53.947001519 +0000 UTC m=+46.604206264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/abaa946e-e127-4ff9-8072-23c540c62eb5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bfh2f" (UID: "abaa946e-e127-4ff9-8072-23c540c62eb5") : configmap "kube-rbac-proxy" not found Jul 13 21:58:52.977378 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.977355 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rnlsb"] Jul 13 21:58:52.977378 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.977391 2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-nj7h6" Jul 13 21:58:52.977589 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.977415 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-nj7h6" Jul 13 21:58:52.977589 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.977488 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rnlsb" Jul 13 21:58:52.980773 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.980753 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6dx77\"" Jul 13 21:58:52.981033 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.981016 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 13 21:58:52.981844 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:52.981822 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 13 21:58:53.012417 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:53.012385 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vpg2f_3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0/konnectivity-agent/0.log" limit=1024 Jul 13 21:58:53.047948 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:53.047913 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch5w2\" (UniqueName: \"kubernetes.io/projected/de451fb9-a1ae-4559-86b7-d643a29152df-kube-api-access-ch5w2\") pod \"migrator-7f59b54f6c-rnlsb\" (UID: \"de451fb9-a1ae-4559-86b7-d643a29152df\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rnlsb" Jul 13 21:58:53.083916 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:53.083881 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vpg2f_3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0/konnectivity-agent/0.log" limit=1024 Jul 13 21:58:53.149074 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:53.149042 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch5w2\" (UniqueName: \"kubernetes.io/projected/de451fb9-a1ae-4559-86b7-d643a29152df-kube-api-access-ch5w2\") pod \"migrator-7f59b54f6c-rnlsb\" (UID: \"de451fb9-a1ae-4559-86b7-d643a29152df\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rnlsb" Jul 13 21:58:53.157873 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:53.157846 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch5w2\" (UniqueName: \"kubernetes.io/projected/de451fb9-a1ae-4559-86b7-d643a29152df-kube-api-access-ch5w2\") pod \"migrator-7f59b54f6c-rnlsb\" (UID: \"de451fb9-a1ae-4559-86b7-d643a29152df\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rnlsb" Jul 13 21:58:53.216652 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:53.216626 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4m9ps_f33a38a4-ac69-4c8f-915f-2e84fa701683/dns-node-resolver/0.log" Jul 13 21:58:53.220192 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:53.220172 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 21:58:53.220491 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:53.220477 2569 scope.go:117] "RemoveContainer" containerID="1f40c7fed419b8ef9c32459c9eeeec0dcf442dba88492d186835a37091160dca" Jul 13 21:58:53.220660 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:53.220639 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-nj7h6_openshift-console-operator(a5e8dab5-94da-46d0-8653-bcc21e849fc1)\"" pod="openshift-console-operator/console-operator-7866497494-nj7h6" podUID="a5e8dab5-94da-46d0-8653-bcc21e849fc1" Jul 13 21:58:53.286263 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:53.286239 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rnlsb" Jul 13 21:58:53.437592 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:53.437513 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rnlsb"] Jul 13 21:58:53.440039 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:53.440013 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde451fb9_a1ae_4559_86b7_d643a29152df.slice/crio-3717ce9ebe24a89870d8c7cdb112053a193250d626e8ebb8f64f166739f50173 WatchSource:0}: Error finding container 3717ce9ebe24a89870d8c7cdb112053a193250d626e8ebb8f64f166739f50173: Status 404 returned error can't find the container with id 3717ce9ebe24a89870d8c7cdb112053a193250d626e8ebb8f64f166739f50173 Jul 13 21:58:53.955148 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:53.955116 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abaa946e-e127-4ff9-8072-23c540c62eb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:53.955289 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:53.955237 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 21:58:53.955325 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:53.955235 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abaa946e-e127-4ff9-8072-23c540c62eb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:53.955325 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:53.955297 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/abaa946e-e127-4ff9-8072-23c540c62eb5-kube-rbac-proxy-cm podName:abaa946e-e127-4ff9-8072-23c540c62eb5 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:55.955278731 +0000 UTC m=+48.612483489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/abaa946e-e127-4ff9-8072-23c540c62eb5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bfh2f" (UID: "abaa946e-e127-4ff9-8072-23c540c62eb5") : configmap "kube-rbac-proxy" not found Jul 13 21:58:53.955325 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:53.955308 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 21:58:53.955435 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:53.955353 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abaa946e-e127-4ff9-8072-23c540c62eb5-insights-runtime-extractor-tls podName:abaa946e-e127-4ff9-8072-23c540c62eb5 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:55.95534198 +0000 UTC m=+48.612546731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abaa946e-e127-4ff9-8072-23c540c62eb5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bfh2f" (UID: "abaa946e-e127-4ff9-8072-23c540c62eb5") : secret "insights-runtime-extractor-tls" not found Jul 13 21:58:54.215809 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.215736 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-782jn_a16c43cb-24ec-420c-953f-204d3584929c/node-ca/0.log" Jul 13 21:58:54.225158 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.225124 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rnlsb" event={"ID":"de451fb9-a1ae-4559-86b7-d643a29152df","Type":"ContainerStarted","Data":"3717ce9ebe24a89870d8c7cdb112053a193250d626e8ebb8f64f166739f50173"} Jul 13 21:58:54.225618 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.225426 2569 scope.go:117] "RemoveContainer" containerID="1f40c7fed419b8ef9c32459c9eeeec0dcf442dba88492d186835a37091160dca" Jul 13 21:58:54.225677 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:54.225630 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-nj7h6_openshift-console-operator(a5e8dab5-94da-46d0-8653-bcc21e849fc1)\"" pod="openshift-console-operator/console-operator-7866497494-nj7h6" podUID="a5e8dab5-94da-46d0-8653-bcc21e849fc1" Jul 13 21:58:54.228797 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.228776 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-ddfdp"] Jul 13 21:58:54.248152 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.248092 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-ddfdp"] Jul 13 21:58:54.248276 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.248217 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-ddfdp" Jul 13 21:58:54.250773 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.250664 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 13 21:58:54.250773 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.250719 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 13 21:58:54.250773 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.250724 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xftpj\"" Jul 13 21:58:54.258639 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.258357 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cf89b421-149a-4fa2-9349-b8b7f2493eaa-nginx-conf\") pod \"networking-console-plugin-6cf856758c-ddfdp\" (UID: \"cf89b421-149a-4fa2-9349-b8b7f2493eaa\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ddfdp" Jul 13 21:58:54.258639 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.258445 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cf89b421-149a-4fa2-9349-b8b7f2493eaa-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ddfdp\" (UID: \"cf89b421-149a-4fa2-9349-b8b7f2493eaa\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ddfdp" Jul 13 21:58:54.346713 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.346672 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vpg2f_3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0/konnectivity-agent/0.log" limit=1024 Jul 13 21:58:54.359652 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.359618 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cf89b421-149a-4fa2-9349-b8b7f2493eaa-nginx-conf\") pod \"networking-console-plugin-6cf856758c-ddfdp\" (UID: \"cf89b421-149a-4fa2-9349-b8b7f2493eaa\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ddfdp" Jul 13 21:58:54.359753 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.359681 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cf89b421-149a-4fa2-9349-b8b7f2493eaa-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ddfdp\" (UID: \"cf89b421-149a-4fa2-9349-b8b7f2493eaa\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ddfdp" Jul 13 21:58:54.359842 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:54.359815 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 21:58:54.359958 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:54.359897 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf89b421-149a-4fa2-9349-b8b7f2493eaa-networking-console-plugin-cert podName:cf89b421-149a-4fa2-9349-b8b7f2493eaa nodeName:}" failed. No retries permitted until 2026-07-13 21:58:54.85987902 +0000 UTC m=+47.517083775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cf89b421-149a-4fa2-9349-b8b7f2493eaa-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-ddfdp" (UID: "cf89b421-149a-4fa2-9349-b8b7f2493eaa") : secret "networking-console-plugin-cert" not found Jul 13 21:58:54.360349 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.360328 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cf89b421-149a-4fa2-9349-b8b7f2493eaa-nginx-conf\") pod \"networking-console-plugin-6cf856758c-ddfdp\" (UID: \"cf89b421-149a-4fa2-9349-b8b7f2493eaa\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ddfdp" Jul 13 21:58:54.448389 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.448359 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vpg2f_3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0/konnectivity-agent/0.log" limit=1024 Jul 13 21:58:54.495495 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.495422 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-9t7dm"] Jul 13 21:58:54.514284 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.514261 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-9t7dm"] Jul 13 21:58:54.514398 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.514360 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-9t7dm" Jul 13 21:58:54.516927 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.516909 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 13 21:58:54.517136 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.517122 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 13 21:58:54.517199 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.517127 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 13 21:58:54.517246 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.517212 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xnspk\"" Jul 13 21:58:54.517501 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.517483 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 13 21:58:54.532513 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.532486 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vpg2f_3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0/konnectivity-agent/0.log" limit=1024 Jul 13 21:58:54.561869 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.561848 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52nh9\" (UniqueName: \"kubernetes.io/projected/86651cf0-a1a5-4543-b724-5267e6877387-kube-api-access-52nh9\") pod \"service-ca-79f68b9f4f-9t7dm\" (UID: \"86651cf0-a1a5-4543-b724-5267e6877387\") " pod="openshift-service-ca/service-ca-79f68b9f4f-9t7dm" Jul 13 21:58:54.561951 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.561914 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/86651cf0-a1a5-4543-b724-5267e6877387-signing-key\") pod \"service-ca-79f68b9f4f-9t7dm\" (UID: \"86651cf0-a1a5-4543-b724-5267e6877387\") " pod="openshift-service-ca/service-ca-79f68b9f4f-9t7dm" Jul 13 21:58:54.561998 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.561959 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/86651cf0-a1a5-4543-b724-5267e6877387-signing-cabundle\") pod \"service-ca-79f68b9f4f-9t7dm\" (UID: \"86651cf0-a1a5-4543-b724-5267e6877387\") " pod="openshift-service-ca/service-ca-79f68b9f4f-9t7dm" Jul 13 21:58:54.590227 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.590192 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vpg2f_3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0/konnectivity-agent/0.log" limit=1024 Jul 13 21:58:54.662314 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.662283 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52nh9\" (UniqueName: \"kubernetes.io/projected/86651cf0-a1a5-4543-b724-5267e6877387-kube-api-access-52nh9\") pod \"service-ca-79f68b9f4f-9t7dm\" (UID: \"86651cf0-a1a5-4543-b724-5267e6877387\") " pod="openshift-service-ca/service-ca-79f68b9f4f-9t7dm" Jul 13 21:58:54.662396 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.662366 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/86651cf0-a1a5-4543-b724-5267e6877387-signing-key\") pod \"service-ca-79f68b9f4f-9t7dm\" (UID: \"86651cf0-a1a5-4543-b724-5267e6877387\") " pod="openshift-service-ca/service-ca-79f68b9f4f-9t7dm" Jul 13 21:58:54.662430 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.662395 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/86651cf0-a1a5-4543-b724-5267e6877387-signing-cabundle\") pod \"service-ca-79f68b9f4f-9t7dm\" (UID: \"86651cf0-a1a5-4543-b724-5267e6877387\") " pod="openshift-service-ca/service-ca-79f68b9f4f-9t7dm" Jul 13 21:58:54.664633 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.664614 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/86651cf0-a1a5-4543-b724-5267e6877387-signing-key\") pod \"service-ca-79f68b9f4f-9t7dm\" (UID: \"86651cf0-a1a5-4543-b724-5267e6877387\") " pod="openshift-service-ca/service-ca-79f68b9f4f-9t7dm" Jul 13 21:58:54.670512 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.670491 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52nh9\" (UniqueName: \"kubernetes.io/projected/86651cf0-a1a5-4543-b724-5267e6877387-kube-api-access-52nh9\") pod \"service-ca-79f68b9f4f-9t7dm\" (UID: \"86651cf0-a1a5-4543-b724-5267e6877387\") " pod="openshift-service-ca/service-ca-79f68b9f4f-9t7dm" Jul 13 21:58:54.670810 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.670790 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/86651cf0-a1a5-4543-b724-5267e6877387-signing-cabundle\") pod \"service-ca-79f68b9f4f-9t7dm\" (UID: \"86651cf0-a1a5-4543-b724-5267e6877387\") " pod="openshift-service-ca/service-ca-79f68b9f4f-9t7dm" Jul 13 21:58:54.822767 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.822700 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-9t7dm" Jul 13 21:58:54.864603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:54.864576 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cf89b421-149a-4fa2-9349-b8b7f2493eaa-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ddfdp\" (UID: \"cf89b421-149a-4fa2-9349-b8b7f2493eaa\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ddfdp" Jul 13 21:58:54.864728 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:54.864677 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 21:58:54.864783 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:54.864750 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf89b421-149a-4fa2-9349-b8b7f2493eaa-networking-console-plugin-cert podName:cf89b421-149a-4fa2-9349-b8b7f2493eaa nodeName:}" failed. No retries permitted until 2026-07-13 21:58:55.864729636 +0000 UTC m=+48.521934421 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cf89b421-149a-4fa2-9349-b8b7f2493eaa-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-ddfdp" (UID: "cf89b421-149a-4fa2-9349-b8b7f2493eaa") : secret "networking-console-plugin-cert" not found Jul 13 21:58:55.016281 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:55.016254 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-thphz_1fee610d-3565-4cfc-8d40-54ab4e09cac8/serve-healthcheck-canary/0.log" Jul 13 21:58:55.072655 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:55.072624 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-9t7dm"] Jul 13 21:58:55.077792 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:58:55.077771 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86651cf0_a1a5_4543_b724_5267e6877387.slice/crio-d012e2bf7d9092fb7982e34562f1da7e2012facdf5a7c69d8a0152d82f760f1e WatchSource:0}: Error finding container d012e2bf7d9092fb7982e34562f1da7e2012facdf5a7c69d8a0152d82f760f1e: Status 404 returned error can't find the container with id d012e2bf7d9092fb7982e34562f1da7e2012facdf5a7c69d8a0152d82f760f1e Jul 13 21:58:55.230017 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:55.229981 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-9t7dm" event={"ID":"86651cf0-a1a5-4543-b724-5267e6877387","Type":"ContainerStarted","Data":"b2d9a69a5c6f4a4e59bbf05b9e21ab3ebfc07f4bd397f5b51048ed90814b2f75"} Jul 13 21:58:55.230370 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:55.230036 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-9t7dm" event={"ID":"86651cf0-a1a5-4543-b724-5267e6877387","Type":"ContainerStarted","Data":"d012e2bf7d9092fb7982e34562f1da7e2012facdf5a7c69d8a0152d82f760f1e"} Jul 13 21:58:55.250639 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:55.250588 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-9t7dm" podStartSLOduration=1.2505726799999999 podStartE2EDuration="1.25057268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:55.249708001 +0000 UTC m=+47.906912777" watchObservedRunningTime="2026-07-13 21:58:55.25057268 +0000 UTC m=+47.907777449" Jul 13 21:58:55.874468 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:55.874430 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cf89b421-149a-4fa2-9349-b8b7f2493eaa-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ddfdp\" (UID: \"cf89b421-149a-4fa2-9349-b8b7f2493eaa\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ddfdp" Jul 13 21:58:55.874652 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:55.874562 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 21:58:55.874652 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:55.874616 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf89b421-149a-4fa2-9349-b8b7f2493eaa-networking-console-plugin-cert podName:cf89b421-149a-4fa2-9349-b8b7f2493eaa nodeName:}" failed. No retries permitted until 2026-07-13 21:58:57.874603519 +0000 UTC m=+50.531808265 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cf89b421-149a-4fa2-9349-b8b7f2493eaa-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-ddfdp" (UID: "cf89b421-149a-4fa2-9349-b8b7f2493eaa") : secret "networking-console-plugin-cert" not found Jul 13 21:58:55.975169 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:55.975142 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abaa946e-e127-4ff9-8072-23c540c62eb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:55.975294 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:55.975260 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 21:58:55.975294 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:55.975265 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abaa946e-e127-4ff9-8072-23c540c62eb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:58:55.975379 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:55.975314 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/abaa946e-e127-4ff9-8072-23c540c62eb5-kube-rbac-proxy-cm podName:abaa946e-e127-4ff9-8072-23c540c62eb5 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:59.975297696 +0000 UTC m=+52.632502465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/abaa946e-e127-4ff9-8072-23c540c62eb5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bfh2f" (UID: "abaa946e-e127-4ff9-8072-23c540c62eb5") : configmap "kube-rbac-proxy" not found Jul 13 21:58:55.975379 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:55.975365 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 21:58:55.975446 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:55.975407 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abaa946e-e127-4ff9-8072-23c540c62eb5-insights-runtime-extractor-tls podName:abaa946e-e127-4ff9-8072-23c540c62eb5 nodeName:}" failed. No retries permitted until 2026-07-13 21:58:59.975395766 +0000 UTC m=+52.632600526 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abaa946e-e127-4ff9-8072-23c540c62eb5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bfh2f" (UID: "abaa946e-e127-4ff9-8072-23c540c62eb5") : secret "insights-runtime-extractor-tls" not found Jul 13 21:58:56.235015 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:56.234982 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rnlsb" event={"ID":"de451fb9-a1ae-4559-86b7-d643a29152df","Type":"ContainerStarted","Data":"8e2e9f2952628509160b2a26b70e6455a5a5c98bc4b5855580845de4ac9a999a"} Jul 13 21:58:56.235366 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:56.235032 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rnlsb" event={"ID":"de451fb9-a1ae-4559-86b7-d643a29152df","Type":"ContainerStarted","Data":"be361ec197061d0822a6f257ba34701ea84620aa808aac73bb44f9fbef668463"} Jul 13 21:58:56.252823 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:56.252777 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rnlsb" podStartSLOduration=2.419482207 podStartE2EDuration="4.252758932s" totalImagesPullingTime="1.833276725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:58:56.251998334 +0000 UTC m=+48.909203103" watchObservedRunningTime="2026-07-13 21:58:56.252758932 +0000 UTC m=+48.909963707" Jul 13 21:58:57.889453 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:57.889416 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cf89b421-149a-4fa2-9349-b8b7f2493eaa-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ddfdp\" (UID: \"cf89b421-149a-4fa2-9349-b8b7f2493eaa\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ddfdp" Jul 13 21:58:57.889907 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:57.889573 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 21:58:57.889907 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:57.889635 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf89b421-149a-4fa2-9349-b8b7f2493eaa-networking-console-plugin-cert podName:cf89b421-149a-4fa2-9349-b8b7f2493eaa nodeName:}" failed. No retries permitted until 2026-07-13 21:59:01.889620022 +0000 UTC m=+54.546824769 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cf89b421-149a-4fa2-9349-b8b7f2493eaa-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-ddfdp" (UID: "cf89b421-149a-4fa2-9349-b8b7f2493eaa") : secret "networking-console-plugin-cert" not found Jul 13 21:58:58.493767 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:58.493732 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:58:58.493923 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:58.493883 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 21:58:58.493923 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:58.493903 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f784dd4c9-l4qbc: secret "image-registry-tls" not found Jul 13 21:58:58.494007 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:58.493956 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls podName:d6cac02d-f5da-4569-894c-0328af666e63 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:14.493942064 +0000 UTC m=+67.151146811 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls") pod "image-registry-7f784dd4c9-l4qbc" (UID: "d6cac02d-f5da-4569-894c-0328af666e63") : secret "image-registry-tls" not found Jul 13 21:58:58.594407 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:58.594379 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30ccc289-8e87-46a2-80b7-064bb0724db7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mscwk\" (UID: \"30ccc289-8e87-46a2-80b7-064bb0724db7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk" Jul 13 21:58:58.594589 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:58.594454 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-stpjr\" (UID: \"779fd208-c240-4e2e-8eed-e76cd9a19023\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" Jul 13 21:58:58.594656 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:58.594624 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 21:58:58.594707 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:58.594694 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls podName:779fd208-c240-4e2e-8eed-e76cd9a19023 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:14.594674958 +0000 UTC m=+67.251879711 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-stpjr" (UID: "779fd208-c240-4e2e-8eed-e76cd9a19023") : secret "cluster-monitoring-operator-tls" not found Jul 13 21:58:58.596654 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:58.596634 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30ccc289-8e87-46a2-80b7-064bb0724db7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mscwk\" (UID: \"30ccc289-8e87-46a2-80b7-064bb0724db7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk" Jul 13 21:58:58.598520 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:58.598492 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk" Jul 13 21:58:58.695603 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:58.695578 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:58.695730 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:58.695609 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:58:58.695730 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:58.695678 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls\") pod \"dns-default-9dkjb\" (UID: \"f8a4145e-0e8f-43f6-83d7-ef573dbc5929\") " pod="openshift-dns/dns-default-9dkjb" Jul 13 21:58:58.695805 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:58.695751 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle podName:d793005d-2131-4e53-8054-703b36d050b8 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:14.695732745 +0000 UTC m=+67.352937512 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle") pod "router-default-768d869684-xh5gb" (UID: "d793005d-2131-4e53-8054-703b36d050b8") : configmap references non-existent config key: service-ca.crt Jul 13 21:58:58.695849 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:58.695807 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 21:58:58.695849 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:58.695842 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls podName:f8a4145e-0e8f-43f6-83d7-ef573dbc5929 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:14.695832001 +0000 UTC m=+67.353036747 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls") pod "dns-default-9dkjb" (UID: "f8a4145e-0e8f-43f6-83d7-ef573dbc5929") : secret "dns-default-metrics-tls" not found Jul 13 21:58:58.695910 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:58.695894 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 21:58:58.695944 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:58:58.695926 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs podName:d793005d-2131-4e53-8054-703b36d050b8 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:14.695916714 +0000 UTC m=+67.353121468 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs") pod "router-default-768d869684-xh5gb" (UID: "d793005d-2131-4e53-8054-703b36d050b8") : secret "router-metrics-certs-default" not found Jul 13 21:58:58.749827 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:58.749759 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk"] Jul 13 21:58:59.245128 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:58:59.245091 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk" event={"ID":"30ccc289-8e87-46a2-80b7-064bb0724db7","Type":"ContainerStarted","Data":"596fa60364825427ff2244b342bf78693a1ff78671211ec5a24d1b0fc368413a"} Jul 13 21:59:00.009479 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:00.009440 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abaa946e-e127-4ff9-8072-23c540c62eb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:59:00.009675 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:00.009589 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abaa946e-e127-4ff9-8072-23c540c62eb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:59:00.009675 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:59:00.009608 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 21:59:00.009675 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:59:00.009674 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abaa946e-e127-4ff9-8072-23c540c62eb5-insights-runtime-extractor-tls podName:abaa946e-e127-4ff9-8072-23c540c62eb5 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:08.009652718 +0000 UTC m=+60.666857465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abaa946e-e127-4ff9-8072-23c540c62eb5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bfh2f" (UID: "abaa946e-e127-4ff9-8072-23c540c62eb5") : secret "insights-runtime-extractor-tls" not found Jul 13 21:59:00.010119 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:00.010096 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abaa946e-e127-4ff9-8072-23c540c62eb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:59:01.252826 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:01.252786 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk" event={"ID":"30ccc289-8e87-46a2-80b7-064bb0724db7","Type":"ContainerStarted","Data":"cdd9a3f247f82d49b567ebe866e6f5e6597565112cb0750c7cdf5444a18dc2d9"} Jul 13 21:59:01.253194 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:01.252841 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk" event={"ID":"30ccc289-8e87-46a2-80b7-064bb0724db7","Type":"ContainerStarted","Data":"f978b1f1e7bbafa3f7ce459ef7ede1ee4272be290fcf4258f69ad391e8081434"} Jul 13 21:59:01.271896 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:01.271841 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mscwk" podStartSLOduration=38.459599466 podStartE2EDuration="40.271821218s" totalImagesPullingTime="1.812221752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:01.271794262 +0000 UTC m=+53.928999028" watchObservedRunningTime="2026-07-13 21:59:01.271821218 +0000 UTC m=+53.929025989" Jul 13 21:59:01.928451 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:01.928419 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cf89b421-149a-4fa2-9349-b8b7f2493eaa-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ddfdp\" (UID: \"cf89b421-149a-4fa2-9349-b8b7f2493eaa\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ddfdp" Jul 13 21:59:01.928685 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:59:01.928580 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 21:59:01.928685 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:59:01.928639 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf89b421-149a-4fa2-9349-b8b7f2493eaa-networking-console-plugin-cert podName:cf89b421-149a-4fa2-9349-b8b7f2493eaa nodeName:}" failed. No retries permitted until 2026-07-13 21:59:09.928625035 +0000 UTC m=+62.585829782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cf89b421-149a-4fa2-9349-b8b7f2493eaa-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-ddfdp" (UID: "cf89b421-149a-4fa2-9349-b8b7f2493eaa") : secret "networking-console-plugin-cert" not found Jul 13 21:59:08.075819 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:08.075785 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abaa946e-e127-4ff9-8072-23c540c62eb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:59:08.078003 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:08.077973 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abaa946e-e127-4ff9-8072-23c540c62eb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bfh2f\" (UID: \"abaa946e-e127-4ff9-8072-23c540c62eb5\") " pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:59:08.176877 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:08.176851 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tlwrc\"" Jul 13 21:59:08.185096 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:08.185080 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bfh2f" Jul 13 21:59:08.307479 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:08.307457 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bfh2f"] Jul 13 21:59:08.309839 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:59:08.309812 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabaa946e_e127_4ff9_8072_23c540c62eb5.slice/crio-0ea99c4b5e030659b20039012174cfb60dc1463a2d958c250bc514eb2a028eb4 WatchSource:0}: Error finding container 0ea99c4b5e030659b20039012174cfb60dc1463a2d958c250bc514eb2a028eb4: Status 404 returned error can't find the container with id 0ea99c4b5e030659b20039012174cfb60dc1463a2d958c250bc514eb2a028eb4 Jul 13 21:59:08.925609 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:08.925571 2569 scope.go:117] "RemoveContainer" containerID="1f40c7fed419b8ef9c32459c9eeeec0dcf442dba88492d186835a37091160dca" Jul 13 21:59:09.286575 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:09.286469 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 21:59:09.287102 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:09.286593 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-nj7h6" event={"ID":"a5e8dab5-94da-46d0-8653-bcc21e849fc1","Type":"ContainerStarted","Data":"e6e9d84923ac77f02655543a2fa80b326fc2031baef59847c859853b2fe0e081"} Jul 13 21:59:09.287102 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:09.286944 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-nj7h6" Jul 13 21:59:09.288299 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:09.288260 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bfh2f" event={"ID":"abaa946e-e127-4ff9-8072-23c540c62eb5","Type":"ContainerStarted","Data":"0ea99c4b5e030659b20039012174cfb60dc1463a2d958c250bc514eb2a028eb4"} Jul 13 21:59:09.307504 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:09.307450 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-nj7h6" podStartSLOduration=40.68744355 podStartE2EDuration="48.307432613s" totalImagesPullingTime="7.619989063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:09.30646625 +0000 UTC m=+61.963671020" watchObservedRunningTime="2026-07-13 21:59:09.307432613 +0000 UTC m=+61.964637382" Jul 13 21:59:09.994194 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:09.994151 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cf89b421-149a-4fa2-9349-b8b7f2493eaa-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ddfdp\" (UID: \"cf89b421-149a-4fa2-9349-b8b7f2493eaa\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ddfdp" Jul 13 21:59:09.997284 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:09.997259 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cf89b421-149a-4fa2-9349-b8b7f2493eaa-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ddfdp\" (UID: \"cf89b421-149a-4fa2-9349-b8b7f2493eaa\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ddfdp" Jul 13 21:59:10.095420 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:10.095391 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9jggp" Jul 13 21:59:10.162076 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:10.162048 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xftpj\"" Jul 13 21:59:10.170184 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:10.170161 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-ddfdp" Jul 13 21:59:10.221399 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:10.221367 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-nj7h6" Jul 13 21:59:12.103824 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:12.103796 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-ddfdp"] Jul 13 21:59:12.107201 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:59:12.107167 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf89b421_149a_4fa2_9349_b8b7f2493eaa.slice/crio-bf7d5c0dc2fb78d28267453cbd7e2246ee8b4f2fce640ed4ad57d190beaa769a WatchSource:0}: Error finding container bf7d5c0dc2fb78d28267453cbd7e2246ee8b4f2fce640ed4ad57d190beaa769a: Status 404 returned error can't find the container with id bf7d5c0dc2fb78d28267453cbd7e2246ee8b4f2fce640ed4ad57d190beaa769a Jul 13 21:59:12.306465 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:12.306374 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bfh2f" event={"ID":"abaa946e-e127-4ff9-8072-23c540c62eb5","Type":"ContainerStarted","Data":"f115c5988886fc73ab273423ddca5af97093d4304a9a0a170e7a2accc60cbc93"} Jul 13 21:59:12.307497 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:12.307467 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-ddfdp" event={"ID":"cf89b421-149a-4fa2-9349-b8b7f2493eaa","Type":"ContainerStarted","Data":"bf7d5c0dc2fb78d28267453cbd7e2246ee8b4f2fce640ed4ad57d190beaa769a"} Jul 13 21:59:13.627228 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:13.627204 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs\") pod \"network-metrics-daemon-tj476\" (UID: \"227fb8d9-c7c7-4722-bba7-f2c2fc9b482f\") " pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:59:13.629959 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:13.629941 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 21:59:13.639401 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:13.639382 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/227fb8d9-c7c7-4722-bba7-f2c2fc9b482f-metrics-certs\") pod \"network-metrics-daemon-tj476\" (UID: \"227fb8d9-c7c7-4722-bba7-f2c2fc9b482f\") " pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:59:13.666454 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:13.666433 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vvmxw\"" Jul 13 21:59:13.674781 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:13.674763 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tj476" Jul 13 21:59:13.727649 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:13.727598 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret\") pod \"global-pull-secret-syncer-qv2xs\" (UID: \"2b1adc79-d0de-44ab-b9c5-747c579b20cb\") " pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:59:13.727649 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:13.727630 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp64d\" (UniqueName: \"kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d\") pod \"network-check-target-qmrjq\" (UID: \"2ee46b9a-7019-44e4-a021-5c4ff5182a65\") " pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:59:13.729884 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:13.729865 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp64d\" (UniqueName: \"kubernetes.io/projected/2ee46b9a-7019-44e4-a021-5c4ff5182a65-kube-api-access-xp64d\") pod \"network-check-target-qmrjq\" (UID: \"2ee46b9a-7019-44e4-a021-5c4ff5182a65\") " pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:59:13.730347 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:13.730332 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 21:59:13.740423 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:13.740402 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b1adc79-d0de-44ab-b9c5-747c579b20cb-original-pull-secret\") pod \"global-pull-secret-syncer-qv2xs\" (UID: \"2b1adc79-d0de-44ab-b9c5-747c579b20cb\") " pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:59:13.905292 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:13.905269 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tj476"] Jul 13 21:59:13.907558 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:59:13.907513 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227fb8d9_c7c7_4722_bba7_f2c2fc9b482f.slice/crio-93558e34e28f8f35c7d2f3021562bc9c58b856b464c86a91b0a80996e7669418 WatchSource:0}: Error finding container 93558e34e28f8f35c7d2f3021562bc9c58b856b464c86a91b0a80996e7669418: Status 404 returned error can't find the container with id 93558e34e28f8f35c7d2f3021562bc9c58b856b464c86a91b0a80996e7669418 Jul 13 21:59:13.945070 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:13.945045 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-djqhc\"" Jul 13 21:59:13.952385 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:13.952360 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:59:13.956701 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:13.956679 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qv2xs" Jul 13 21:59:14.105186 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.105158 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qmrjq"] Jul 13 21:59:14.107944 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:59:14.107913 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee46b9a_7019_44e4_a021_5c4ff5182a65.slice/crio-810540fd2cc5ad0de70cebde7e2c7a40390e3cd74f23609a657e053b3019073a WatchSource:0}: Error finding container 810540fd2cc5ad0de70cebde7e2c7a40390e3cd74f23609a657e053b3019073a: Status 404 returned error can't find the container with id 810540fd2cc5ad0de70cebde7e2c7a40390e3cd74f23609a657e053b3019073a Jul 13 21:59:14.123421 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.123256 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qv2xs"] Jul 13 21:59:14.125758 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:59:14.125733 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1adc79_d0de_44ab_b9c5_747c579b20cb.slice/crio-62cdf256845a8118f2a000f169d5e7550e8793cb2ae7d972488488624cba37e7 WatchSource:0}: Error finding container 62cdf256845a8118f2a000f169d5e7550e8793cb2ae7d972488488624cba37e7: Status 404 returned error can't find the container with id 62cdf256845a8118f2a000f169d5e7550e8793cb2ae7d972488488624cba37e7 Jul 13 21:59:14.320741 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.317124 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-ddfdp" event={"ID":"cf89b421-149a-4fa2-9349-b8b7f2493eaa","Type":"ContainerStarted","Data":"96f41a6f6c858913f42a0e6a9c50b491692bcc944f2f355aa599b58fb866c763"} Jul 13 21:59:14.320741 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.318790 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qv2xs" event={"ID":"2b1adc79-d0de-44ab-b9c5-747c579b20cb","Type":"ContainerStarted","Data":"62cdf256845a8118f2a000f169d5e7550e8793cb2ae7d972488488624cba37e7"} Jul 13 21:59:14.322516 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.322483 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tj476" event={"ID":"227fb8d9-c7c7-4722-bba7-f2c2fc9b482f","Type":"ContainerStarted","Data":"93558e34e28f8f35c7d2f3021562bc9c58b856b464c86a91b0a80996e7669418"} Jul 13 21:59:14.324429 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.324403 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bfh2f" event={"ID":"abaa946e-e127-4ff9-8072-23c540c62eb5","Type":"ContainerStarted","Data":"2e32027d45e12e170f20fc6ef741c8cc5fdacd5444a3ad92f6d4c676d7a83ac8"} Jul 13 21:59:14.326142 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.326106 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qmrjq" event={"ID":"2ee46b9a-7019-44e4-a021-5c4ff5182a65","Type":"ContainerStarted","Data":"be50771c29149a103885fb3b933b0146b109914be2d9d8938f9d5c077170eba9"} Jul 13 21:59:14.326249 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.326148 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qmrjq" event={"ID":"2ee46b9a-7019-44e4-a021-5c4ff5182a65","Type":"ContainerStarted","Data":"810540fd2cc5ad0de70cebde7e2c7a40390e3cd74f23609a657e053b3019073a"} Jul 13 21:59:14.326319 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.326256 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:59:14.334082 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.334034 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-ddfdp" podStartSLOduration=18.683013852 podStartE2EDuration="20.334022504s" totalImagesPullingTime="1.651008652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:14.333347165 +0000 UTC m=+66.990551934" watchObservedRunningTime="2026-07-13 21:59:14.334022504 +0000 UTC m=+66.991227316" Jul 13 21:59:14.348768 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.348726 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qmrjq" podStartSLOduration=66.348711892 podStartE2EDuration="1m6.348711892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:14.348321698 +0000 UTC m=+67.005526468" watchObservedRunningTime="2026-07-13 21:59:14.348711892 +0000 UTC m=+67.005916653" Jul 13 21:59:14.536437 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.536396 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:59:14.539377 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.539341 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls\") pod \"image-registry-7f784dd4c9-l4qbc\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:59:14.637195 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.637100 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-stpjr\" (UID: \"779fd208-c240-4e2e-8eed-e76cd9a19023\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" Jul 13 21:59:14.644993 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.644967 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/779fd208-c240-4e2e-8eed-e76cd9a19023-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-stpjr\" (UID: \"779fd208-c240-4e2e-8eed-e76cd9a19023\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" Jul 13 21:59:14.738051 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.738014 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:59:14.738220 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.738059 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:59:14.738220 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.738118 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls\") pod \"dns-default-9dkjb\" (UID: \"f8a4145e-0e8f-43f6-83d7-ef573dbc5929\") " pod="openshift-dns/dns-default-9dkjb" Jul 13 21:59:14.738860 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.738813 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d793005d-2131-4e53-8054-703b36d050b8-service-ca-bundle\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:59:14.740867 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.740824 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d793005d-2131-4e53-8054-703b36d050b8-metrics-certs\") pod \"router-default-768d869684-xh5gb\" (UID: \"d793005d-2131-4e53-8054-703b36d050b8\") " pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:59:14.741439 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.741417 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8a4145e-0e8f-43f6-83d7-ef573dbc5929-metrics-tls\") pod \"dns-default-9dkjb\" (UID: \"f8a4145e-0e8f-43f6-83d7-ef573dbc5929\") " pod="openshift-dns/dns-default-9dkjb" Jul 13 21:59:14.777941 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.777914 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p9jhm\"" Jul 13 21:59:14.785619 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.785556 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:59:14.844393 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.844366 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9vd8k\"" Jul 13 21:59:14.852753 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.852532 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" Jul 13 21:59:14.888881 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.888789 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jqfwq\"" Jul 13 21:59:14.896985 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.896875 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9dkjb" Jul 13 21:59:14.915997 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.915821 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-l7jtp\"" Jul 13 21:59:14.923303 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:14.923271 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:59:16.419578 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:16.419424 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9dkjb"] Jul 13 21:59:16.423114 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:59:16.423083 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a4145e_0e8f_43f6_83d7_ef573dbc5929.slice/crio-c082cbe190abd3d3cbb899d26e698b408297a006568701589a14e141a8ad720e WatchSource:0}: Error finding container c082cbe190abd3d3cbb899d26e698b408297a006568701589a14e141a8ad720e: Status 404 returned error can't find the container with id c082cbe190abd3d3cbb899d26e698b408297a006568701589a14e141a8ad720e Jul 13 21:59:16.443967 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:16.443833 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-768d869684-xh5gb"] Jul 13 21:59:16.450940 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:59:16.450863 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd793005d_2131_4e53_8054_703b36d050b8.slice/crio-a2b395a65add63bb692705fb181e880342a154cbf3df8dd00f9f3355c163e650 WatchSource:0}: Error finding container a2b395a65add63bb692705fb181e880342a154cbf3df8dd00f9f3355c163e650: Status 404 returned error can't find the container with id a2b395a65add63bb692705fb181e880342a154cbf3df8dd00f9f3355c163e650 Jul 13 21:59:16.668995 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:16.668963 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr"] Jul 13 21:59:16.671917 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:59:16.671886 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779fd208_c240_4e2e_8eed_e76cd9a19023.slice/crio-20c1b3c1f4ed538c2303345905defba5664f19b3972a41497c45036795fa203e WatchSource:0}: Error finding container 20c1b3c1f4ed538c2303345905defba5664f19b3972a41497c45036795fa203e: Status 404 returned error can't find the container with id 20c1b3c1f4ed538c2303345905defba5664f19b3972a41497c45036795fa203e Jul 13 21:59:16.672228 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:16.672204 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f784dd4c9-l4qbc"] Jul 13 21:59:16.675026 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:59:16.675001 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6cac02d_f5da_4569_894c_0328af666e63.slice/crio-647ab0ae1676010b639919fe599e6a42a69c56ca763df63030021d0f37a28bc4 WatchSource:0}: Error finding container 647ab0ae1676010b639919fe599e6a42a69c56ca763df63030021d0f37a28bc4: Status 404 returned error can't find the container with id 647ab0ae1676010b639919fe599e6a42a69c56ca763df63030021d0f37a28bc4 Jul 13 21:59:17.348582 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:17.348325 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" event={"ID":"d6cac02d-f5da-4569-894c-0328af666e63","Type":"ContainerStarted","Data":"15bbe1b22f1956c1b56b5141300cf23dee814e873f26a3375e343b0e437deb69"} Jul 13 21:59:17.348582 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:17.348383 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" event={"ID":"d6cac02d-f5da-4569-894c-0328af666e63","Type":"ContainerStarted","Data":"647ab0ae1676010b639919fe599e6a42a69c56ca763df63030021d0f37a28bc4"} Jul 13 21:59:17.348816 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:17.348608 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:59:17.349993 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:17.349964 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9dkjb" event={"ID":"f8a4145e-0e8f-43f6-83d7-ef573dbc5929","Type":"ContainerStarted","Data":"c082cbe190abd3d3cbb899d26e698b408297a006568701589a14e141a8ad720e"} Jul 13 21:59:17.351768 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:17.351740 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-768d869684-xh5gb" event={"ID":"d793005d-2131-4e53-8054-703b36d050b8","Type":"ContainerStarted","Data":"9ff18efe82c3283e97e8a0e13c1fe33ef058e90eb6bdc22be38e652c1d68073d"} Jul 13 21:59:17.351879 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:17.351794 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-768d869684-xh5gb" event={"ID":"d793005d-2131-4e53-8054-703b36d050b8","Type":"ContainerStarted","Data":"a2b395a65add63bb692705fb181e880342a154cbf3df8dd00f9f3355c163e650"} Jul 13 21:59:17.353358 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:17.353311 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" event={"ID":"779fd208-c240-4e2e-8eed-e76cd9a19023","Type":"ContainerStarted","Data":"20c1b3c1f4ed538c2303345905defba5664f19b3972a41497c45036795fa203e"} Jul 13 21:59:17.356003 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:17.355973 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bfh2f" event={"ID":"abaa946e-e127-4ff9-8072-23c540c62eb5","Type":"ContainerStarted","Data":"84acc14802d39a1616352981ca2d8a27952d944801d24ac5731add6fe59f2724"} Jul 13 21:59:17.358225 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:17.358200 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tj476" event={"ID":"227fb8d9-c7c7-4722-bba7-f2c2fc9b482f","Type":"ContainerStarted","Data":"fbaa04be9275bdcdbda61028c5e0adea534c12c8a054f97b620210de3eec52ee"} Jul 13 21:59:17.358310 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:17.358248 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tj476" event={"ID":"227fb8d9-c7c7-4722-bba7-f2c2fc9b482f","Type":"ContainerStarted","Data":"916ca1c17269ce11d3530eec75c8c59575170c1c612dea32fcdd743e8c7eeb29"} Jul 13 21:59:17.371131 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:17.370651 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" podStartSLOduration=69.370639099 podStartE2EDuration="1m9.370639099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:17.370217657 +0000 UTC m=+70.027422427" watchObservedRunningTime="2026-07-13 21:59:17.370639099 +0000 UTC m=+70.027843867" Jul 13 21:59:17.388432 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:17.388374 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-768d869684-xh5gb" podStartSLOduration=56.3883564 podStartE2EDuration="56.3883564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:17.388141178 +0000 UTC m=+70.045345947" watchObservedRunningTime="2026-07-13 21:59:17.3883564 +0000 UTC m=+70.045561169" Jul 13 21:59:17.405517 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:17.404875 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tj476" podStartSLOduration=67.073877123 podStartE2EDuration="1m9.404857263s" totalImagesPullingTime="2.33098014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:17.403877377 +0000 UTC m=+70.061082145" watchObservedRunningTime="2026-07-13 21:59:17.404857263 +0000 UTC m=+70.062062034" Jul 13 21:59:17.421855 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:17.421810 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bfh2f" podStartSLOduration=17.66096395 podStartE2EDuration="25.421794784s" totalImagesPullingTime="7.760830834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:58:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:17.421099342 +0000 UTC m=+70.078304110" watchObservedRunningTime="2026-07-13 21:59:17.421794784 +0000 UTC m=+70.078999554" Jul 13 21:59:17.923643 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:17.923576 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:59:17.936073 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:17.936046 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:59:18.363715 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:18.363659 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:59:18.364610 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:18.364584 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-768d869684-xh5gb" Jul 13 21:59:19.305349 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.305313 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c788876d8-ww7qf"] Jul 13 21:59:19.311447 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.311424 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv"] Jul 13 21:59:19.311595 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.311552 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c788876d8-ww7qf" Jul 13 21:59:19.314668 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.314640 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 13 21:59:19.314793 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.314690 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749684677d-rdjxl"] Jul 13 21:59:19.314793 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.314640 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 13 21:59:19.314904 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.314809 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.314904 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.314640 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 13 21:59:19.315330 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.315309 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 13 21:59:19.317218 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.317201 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 13 21:59:19.317425 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.317406 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749684677d-rdjxl" Jul 13 21:59:19.317645 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.317627 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 13 21:59:19.317719 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.317696 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 13 21:59:19.317864 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.317841 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 13 21:59:19.320437 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.320395 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c788876d8-ww7qf"] Jul 13 21:59:19.321800 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.320936 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 13 21:59:19.321800 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.321210 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-f2ph4\"" Jul 13 21:59:19.323908 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.323868 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv"] Jul 13 21:59:19.326988 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.326965 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749684677d-rdjxl"] Jul 13 21:59:19.383325 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.383283 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cf4c85ba-e813-4696-a51a-a6b8a353a294-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77cb794d48-pmrmv\" (UID: \"cf4c85ba-e813-4696-a51a-a6b8a353a294\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.383503 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.383331 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/259a215e-706d-4578-99cc-7f236307874c-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c788876d8-ww7qf\" (UID: \"259a215e-706d-4578-99cc-7f236307874c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c788876d8-ww7qf" Jul 13 21:59:19.383503 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.383363 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cf4c85ba-e813-4696-a51a-a6b8a353a294-hub\") pod \"cluster-proxy-proxy-agent-77cb794d48-pmrmv\" (UID: \"cf4c85ba-e813-4696-a51a-a6b8a353a294\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.383503 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.383389 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4c85ba-e813-4696-a51a-a6b8a353a294-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77cb794d48-pmrmv\" (UID: \"cf4c85ba-e813-4696-a51a-a6b8a353a294\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.383503 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.383412 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4f2w\" (UniqueName: \"kubernetes.io/projected/259a215e-706d-4578-99cc-7f236307874c-kube-api-access-f4f2w\") pod \"klusterlet-addon-workmgr-7c788876d8-ww7qf\" (UID: \"259a215e-706d-4578-99cc-7f236307874c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c788876d8-ww7qf" Jul 13 21:59:19.383773 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.383510 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/259a215e-706d-4578-99cc-7f236307874c-tmp\") pod \"klusterlet-addon-workmgr-7c788876d8-ww7qf\" (UID: \"259a215e-706d-4578-99cc-7f236307874c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c788876d8-ww7qf" Jul 13 21:59:19.383773 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.383604 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mn57\" (UniqueName: \"kubernetes.io/projected/cf4c85ba-e813-4696-a51a-a6b8a353a294-kube-api-access-9mn57\") pod \"cluster-proxy-proxy-agent-77cb794d48-pmrmv\" (UID: \"cf4c85ba-e813-4696-a51a-a6b8a353a294\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.383773 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.383671 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cf4c85ba-e813-4696-a51a-a6b8a353a294-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77cb794d48-pmrmv\" (UID: \"cf4c85ba-e813-4696-a51a-a6b8a353a294\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.383773 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.383711 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cf4c85ba-e813-4696-a51a-a6b8a353a294-ca\") pod \"cluster-proxy-proxy-agent-77cb794d48-pmrmv\" (UID: \"cf4c85ba-e813-4696-a51a-a6b8a353a294\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.415237 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.415204 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-78th4"] Jul 13 21:59:19.419092 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.419063 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-78th4" Jul 13 21:59:19.422245 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.422223 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cfc77bcff-76rjs"] Jul 13 21:59:19.425581 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.425557 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.426457 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.426434 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gmtng\"" Jul 13 21:59:19.426786 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.426761 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 13 21:59:19.427327 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.427301 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 13 21:59:19.428228 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.428211 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 13 21:59:19.428634 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.428481 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 13 21:59:19.428634 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.428493 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 13 21:59:19.428634 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.428472 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 13 21:59:19.430236 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.430205 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 13 21:59:19.431342 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.431320 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d95cx\"" Jul 13 21:59:19.436626 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.436601 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-78th4"] Jul 13 21:59:19.444660 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.444618 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cfc77bcff-76rjs"] Jul 13 21:59:19.485766 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.484918 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cf4c85ba-e813-4696-a51a-a6b8a353a294-hub\") pod \"cluster-proxy-proxy-agent-77cb794d48-pmrmv\" (UID: \"cf4c85ba-e813-4696-a51a-a6b8a353a294\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.485766 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.484967 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4c85ba-e813-4696-a51a-a6b8a353a294-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77cb794d48-pmrmv\" (UID: \"cf4c85ba-e813-4696-a51a-a6b8a353a294\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.485766 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.484995 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4f2w\" (UniqueName: \"kubernetes.io/projected/259a215e-706d-4578-99cc-7f236307874c-kube-api-access-f4f2w\") pod \"klusterlet-addon-workmgr-7c788876d8-ww7qf\" (UID: \"259a215e-706d-4578-99cc-7f236307874c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c788876d8-ww7qf" Jul 13 21:59:19.485766 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.485062 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh54j\" (UniqueName: \"kubernetes.io/projected/c541c830-4b9b-43a1-8351-222c81fa2368-kube-api-access-rh54j\") pod \"managed-serviceaccount-addon-agent-749684677d-rdjxl\" (UID: \"c541c830-4b9b-43a1-8351-222c81fa2368\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749684677d-rdjxl" Jul 13 21:59:19.485766 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.485099 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/259a215e-706d-4578-99cc-7f236307874c-tmp\") pod \"klusterlet-addon-workmgr-7c788876d8-ww7qf\" (UID: \"259a215e-706d-4578-99cc-7f236307874c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c788876d8-ww7qf" Jul 13 21:59:19.485766 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.485189 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mn57\" (UniqueName: \"kubernetes.io/projected/cf4c85ba-e813-4696-a51a-a6b8a353a294-kube-api-access-9mn57\") pod \"cluster-proxy-proxy-agent-77cb794d48-pmrmv\" (UID: \"cf4c85ba-e813-4696-a51a-a6b8a353a294\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.485766 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.485214 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c541c830-4b9b-43a1-8351-222c81fa2368-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-749684677d-rdjxl\" (UID: \"c541c830-4b9b-43a1-8351-222c81fa2368\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749684677d-rdjxl" Jul 13 21:59:19.485766 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.485271 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cf4c85ba-e813-4696-a51a-a6b8a353a294-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77cb794d48-pmrmv\" (UID: \"cf4c85ba-e813-4696-a51a-a6b8a353a294\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.485766 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.485306 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cf4c85ba-e813-4696-a51a-a6b8a353a294-ca\") pod \"cluster-proxy-proxy-agent-77cb794d48-pmrmv\" (UID: \"cf4c85ba-e813-4696-a51a-a6b8a353a294\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.485766 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.485348 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cf4c85ba-e813-4696-a51a-a6b8a353a294-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77cb794d48-pmrmv\" (UID: \"cf4c85ba-e813-4696-a51a-a6b8a353a294\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.485766 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.485381 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/259a215e-706d-4578-99cc-7f236307874c-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c788876d8-ww7qf\" (UID: \"259a215e-706d-4578-99cc-7f236307874c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c788876d8-ww7qf" Jul 13 21:59:19.487093 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.487052 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/259a215e-706d-4578-99cc-7f236307874c-tmp\") pod \"klusterlet-addon-workmgr-7c788876d8-ww7qf\" (UID: \"259a215e-706d-4578-99cc-7f236307874c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c788876d8-ww7qf" Jul 13 21:59:19.487215 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.487187 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4c85ba-e813-4696-a51a-a6b8a353a294-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77cb794d48-pmrmv\" (UID: \"cf4c85ba-e813-4696-a51a-a6b8a353a294\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.488817 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.488770 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/259a215e-706d-4578-99cc-7f236307874c-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c788876d8-ww7qf\" (UID: \"259a215e-706d-4578-99cc-7f236307874c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c788876d8-ww7qf" Jul 13 21:59:19.489363 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.489335 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cf4c85ba-e813-4696-a51a-a6b8a353a294-hub\") pod \"cluster-proxy-proxy-agent-77cb794d48-pmrmv\" (UID: \"cf4c85ba-e813-4696-a51a-a6b8a353a294\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.489665 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.489640 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cf4c85ba-e813-4696-a51a-a6b8a353a294-ca\") pod \"cluster-proxy-proxy-agent-77cb794d48-pmrmv\" (UID: \"cf4c85ba-e813-4696-a51a-a6b8a353a294\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.490080 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.490057 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cf4c85ba-e813-4696-a51a-a6b8a353a294-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77cb794d48-pmrmv\" (UID: \"cf4c85ba-e813-4696-a51a-a6b8a353a294\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.490080 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.490074 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cf4c85ba-e813-4696-a51a-a6b8a353a294-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77cb794d48-pmrmv\" (UID: \"cf4c85ba-e813-4696-a51a-a6b8a353a294\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.503600 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.503572 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4f2w\" (UniqueName: \"kubernetes.io/projected/259a215e-706d-4578-99cc-7f236307874c-kube-api-access-f4f2w\") pod \"klusterlet-addon-workmgr-7c788876d8-ww7qf\" (UID: \"259a215e-706d-4578-99cc-7f236307874c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c788876d8-ww7qf" Jul 13 21:59:19.503926 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.503880 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mn57\" (UniqueName: \"kubernetes.io/projected/cf4c85ba-e813-4696-a51a-a6b8a353a294-kube-api-access-9mn57\") pod \"cluster-proxy-proxy-agent-77cb794d48-pmrmv\" (UID: \"cf4c85ba-e813-4696-a51a-a6b8a353a294\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.586334 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.586258 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-console-oauth-config\") pod \"console-cfc77bcff-76rjs\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.586334 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.586303 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-service-ca\") pod \"console-cfc77bcff-76rjs\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.586523 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.586369 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q646r\" (UniqueName: \"kubernetes.io/projected/49ff524c-8e50-4f1e-adf4-778791a3191e-kube-api-access-q646r\") pod \"downloads-68c9cbb497-78th4\" (UID: \"49ff524c-8e50-4f1e-adf4-778791a3191e\") " pod="openshift-console/downloads-68c9cbb497-78th4" Jul 13 21:59:19.586523 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.586400 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-oauth-serving-cert\") pod \"console-cfc77bcff-76rjs\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.586523 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.586450 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-console-config\") pod \"console-cfc77bcff-76rjs\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.586523 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.586511 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh54j\" (UniqueName: \"kubernetes.io/projected/c541c830-4b9b-43a1-8351-222c81fa2368-kube-api-access-rh54j\") pod \"managed-serviceaccount-addon-agent-749684677d-rdjxl\" (UID: \"c541c830-4b9b-43a1-8351-222c81fa2368\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749684677d-rdjxl" Jul 13 21:59:19.586739 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.586564 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f6w8\" (UniqueName: \"kubernetes.io/projected/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-kube-api-access-6f6w8\") pod \"console-cfc77bcff-76rjs\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.586739 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.586590 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-console-serving-cert\") pod \"console-cfc77bcff-76rjs\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.586739 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.586637 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c541c830-4b9b-43a1-8351-222c81fa2368-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-749684677d-rdjxl\" (UID: \"c541c830-4b9b-43a1-8351-222c81fa2368\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749684677d-rdjxl" Jul 13 21:59:19.589454 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.589428 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c541c830-4b9b-43a1-8351-222c81fa2368-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-749684677d-rdjxl\" (UID: \"c541c830-4b9b-43a1-8351-222c81fa2368\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749684677d-rdjxl" Jul 13 21:59:19.595213 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.595189 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh54j\" (UniqueName: \"kubernetes.io/projected/c541c830-4b9b-43a1-8351-222c81fa2368-kube-api-access-rh54j\") pod \"managed-serviceaccount-addon-agent-749684677d-rdjxl\" (UID: \"c541c830-4b9b-43a1-8351-222c81fa2368\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749684677d-rdjxl" Jul 13 21:59:19.627549 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.627490 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c788876d8-ww7qf" Jul 13 21:59:19.639550 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.639512 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" Jul 13 21:59:19.646660 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.646636 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749684677d-rdjxl" Jul 13 21:59:19.687047 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.687018 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-oauth-serving-cert\") pod \"console-cfc77bcff-76rjs\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.687151 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.687057 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-console-config\") pod \"console-cfc77bcff-76rjs\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.687151 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.687104 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f6w8\" (UniqueName: \"kubernetes.io/projected/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-kube-api-access-6f6w8\") pod \"console-cfc77bcff-76rjs\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.687328 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.687304 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-console-serving-cert\") pod \"console-cfc77bcff-76rjs\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.687438 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.687394 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-console-oauth-config\") pod \"console-cfc77bcff-76rjs\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.687438 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.687426 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-service-ca\") pod \"console-cfc77bcff-76rjs\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.687572 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.687465 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q646r\" (UniqueName: \"kubernetes.io/projected/49ff524c-8e50-4f1e-adf4-778791a3191e-kube-api-access-q646r\") pod \"downloads-68c9cbb497-78th4\" (UID: \"49ff524c-8e50-4f1e-adf4-778791a3191e\") " pod="openshift-console/downloads-68c9cbb497-78th4" Jul 13 21:59:19.687826 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.687795 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-oauth-serving-cert\") pod \"console-cfc77bcff-76rjs\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.688304 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.688280 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-console-config\") pod \"console-cfc77bcff-76rjs\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.688424 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.688406 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-service-ca\") pod \"console-cfc77bcff-76rjs\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.690002 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.689957 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-console-serving-cert\") pod \"console-cfc77bcff-76rjs\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.690242 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.690224 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-console-oauth-config\") pod \"console-cfc77bcff-76rjs\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.699414 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.699396 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q646r\" (UniqueName: \"kubernetes.io/projected/49ff524c-8e50-4f1e-adf4-778791a3191e-kube-api-access-q646r\") pod \"downloads-68c9cbb497-78th4\" (UID: \"49ff524c-8e50-4f1e-adf4-778791a3191e\") " pod="openshift-console/downloads-68c9cbb497-78th4" Jul 13 21:59:19.700078 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.700059 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f6w8\" (UniqueName: \"kubernetes.io/projected/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-kube-api-access-6f6w8\") pod \"console-cfc77bcff-76rjs\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.748174 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.748150 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-78th4" Jul 13 21:59:19.755276 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.755257 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:19.930249 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.930171 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b445969f5-p5986"] Jul 13 21:59:19.949527 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.949498 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b445969f5-p5986"] Jul 13 21:59:19.949690 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.949599 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:19.959234 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:19.959213 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 13 21:59:20.090294 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.090268 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpfw8\" (UniqueName: \"kubernetes.io/projected/c749ca98-5b06-4ff7-9282-3c72a1d8b036-kube-api-access-tpfw8\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.090406 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.090307 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c749ca98-5b06-4ff7-9282-3c72a1d8b036-console-serving-cert\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.090406 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.090326 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-trusted-ca-bundle\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.090406 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.090394 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-service-ca\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.090530 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.090422 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c749ca98-5b06-4ff7-9282-3c72a1d8b036-console-oauth-config\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.090530 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.090442 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-oauth-serving-cert\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.090530 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.090464 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-console-config\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.191532 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.191503 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-service-ca\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.191659 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.191580 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c749ca98-5b06-4ff7-9282-3c72a1d8b036-console-oauth-config\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.191659 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.191607 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-oauth-serving-cert\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.191659 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.191637 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-console-config\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.191813 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.191684 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpfw8\" (UniqueName: \"kubernetes.io/projected/c749ca98-5b06-4ff7-9282-3c72a1d8b036-kube-api-access-tpfw8\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.191813 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.191742 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c749ca98-5b06-4ff7-9282-3c72a1d8b036-console-serving-cert\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.191813 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.191778 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-trusted-ca-bundle\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.192221 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.192201 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-service-ca\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.192322 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.192301 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-oauth-serving-cert\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.192413 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.192395 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-console-config\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.192569 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.192531 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-trusted-ca-bundle\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.193927 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.193906 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c749ca98-5b06-4ff7-9282-3c72a1d8b036-console-oauth-config\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.194045 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.194026 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c749ca98-5b06-4ff7-9282-3c72a1d8b036-console-serving-cert\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.199228 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.199210 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpfw8\" (UniqueName: \"kubernetes.io/projected/c749ca98-5b06-4ff7-9282-3c72a1d8b036-kube-api-access-tpfw8\") pod \"console-6b445969f5-p5986\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.259600 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.259529 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:20.441097 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.441015 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-78th4"] Jul 13 21:59:20.460579 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:59:20.460411 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ff524c_8e50_4f1e_adf4_778791a3191e.slice/crio-0c2748d2872f9012febceec63fed91ede750d0ff25e327c9281791270fadbe7c WatchSource:0}: Error finding container 0c2748d2872f9012febceec63fed91ede750d0ff25e327c9281791270fadbe7c: Status 404 returned error can't find the container with id 0c2748d2872f9012febceec63fed91ede750d0ff25e327c9281791270fadbe7c Jul 13 21:59:20.685123 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.685047 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c788876d8-ww7qf"] Jul 13 21:59:20.691080 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.691051 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cfc77bcff-76rjs"] Jul 13 21:59:20.692071 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:59:20.691986 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259a215e_706d_4578_99cc_7f236307874c.slice/crio-b5b03c6d19c43e9be8ce3cf0552eec4ecf1ec6e2d8a3b6b37aaee62af942b2db WatchSource:0}: Error finding container b5b03c6d19c43e9be8ce3cf0552eec4ecf1ec6e2d8a3b6b37aaee62af942b2db: Status 404 returned error can't find the container with id b5b03c6d19c43e9be8ce3cf0552eec4ecf1ec6e2d8a3b6b37aaee62af942b2db Jul 13 21:59:20.695083 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:59:20.695059 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ee462f_da9d_40ec_a8b7_b1f49e79174c.slice/crio-772233439bed21aafb1f565172234d4e0f91e38667b6b55106815254c788753c WatchSource:0}: Error finding container 772233439bed21aafb1f565172234d4e0f91e38667b6b55106815254c788753c: Status 404 returned error can't find the container with id 772233439bed21aafb1f565172234d4e0f91e38667b6b55106815254c788753c Jul 13 21:59:20.698819 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.698798 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749684677d-rdjxl"] Jul 13 21:59:20.748591 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.748567 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv"] Jul 13 21:59:20.750927 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:59:20.750904 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf4c85ba_e813_4696_a51a_a6b8a353a294.slice/crio-944a8154ee9d819a5b9b0c7d236b6ece4d653d5519a11113ebe48173d96bd690 WatchSource:0}: Error finding container 944a8154ee9d819a5b9b0c7d236b6ece4d653d5519a11113ebe48173d96bd690: Status 404 returned error can't find the container with id 944a8154ee9d819a5b9b0c7d236b6ece4d653d5519a11113ebe48173d96bd690 Jul 13 21:59:20.840605 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:20.840573 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b445969f5-p5986"] Jul 13 21:59:20.843647 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:59:20.843620 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc749ca98_5b06_4ff7_9282_3c72a1d8b036.slice/crio-2dcc0be614b10d832b3cbe7ac0345ea679586c2106fe43a51cfb8342e7f916b6 WatchSource:0}: Error finding container 2dcc0be614b10d832b3cbe7ac0345ea679586c2106fe43a51cfb8342e7f916b6: Status 404 returned error can't find the container with id 2dcc0be614b10d832b3cbe7ac0345ea679586c2106fe43a51cfb8342e7f916b6 Jul 13 21:59:21.001108 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.001023 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pqh9"] Jul 13 21:59:21.034589 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.034529 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pqh9"] Jul 13 21:59:21.034760 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.034630 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pqh9" Jul 13 21:59:21.037129 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.037109 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 13 21:59:21.037294 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.037200 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-l4qg8\"" Jul 13 21:59:21.202872 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.202836 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/557b4511-3dad-4e13-92cb-0d5c4eef6af5-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-6pqh9\" (UID: \"557b4511-3dad-4e13-92cb-0d5c4eef6af5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pqh9" Jul 13 21:59:21.303779 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.303682 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/557b4511-3dad-4e13-92cb-0d5c4eef6af5-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-6pqh9\" (UID: \"557b4511-3dad-4e13-92cb-0d5c4eef6af5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pqh9" Jul 13 21:59:21.306422 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.306396 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/557b4511-3dad-4e13-92cb-0d5c4eef6af5-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-6pqh9\" (UID: \"557b4511-3dad-4e13-92cb-0d5c4eef6af5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pqh9" Jul 13 21:59:21.346318 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.346289 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pqh9" Jul 13 21:59:21.415573 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.415450 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9dkjb" event={"ID":"f8a4145e-0e8f-43f6-83d7-ef573dbc5929","Type":"ContainerStarted","Data":"c51a14983948593403148b5042fd84669e34872aec4e715f41089475220da6c5"} Jul 13 21:59:21.415573 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.415514 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9dkjb" event={"ID":"f8a4145e-0e8f-43f6-83d7-ef573dbc5929","Type":"ContainerStarted","Data":"d1bd0b8d9208f30406eb9eb1d977dc8f0612571ec12408d22f7316258b870aad"} Jul 13 21:59:21.416068 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.416012 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9dkjb" Jul 13 21:59:21.417650 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.417585 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c788876d8-ww7qf" event={"ID":"259a215e-706d-4578-99cc-7f236307874c","Type":"ContainerStarted","Data":"b5b03c6d19c43e9be8ce3cf0552eec4ecf1ec6e2d8a3b6b37aaee62af942b2db"} Jul 13 21:59:21.420310 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.420256 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749684677d-rdjxl" event={"ID":"c541c830-4b9b-43a1-8351-222c81fa2368","Type":"ContainerStarted","Data":"fcafeabc970181c2697ad2c5dbb1af966df8952df40f7d4c316b135fddc5ccf1"} Jul 13 21:59:21.423242 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.423187 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" event={"ID":"779fd208-c240-4e2e-8eed-e76cd9a19023","Type":"ContainerStarted","Data":"897ca2207e33b73a9b0ae73b4d1c9affc858f131a388c445a77bacd01d15917d"} Jul 13 21:59:21.425975 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.425923 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" event={"ID":"cf4c85ba-e813-4696-a51a-a6b8a353a294","Type":"ContainerStarted","Data":"944a8154ee9d819a5b9b0c7d236b6ece4d653d5519a11113ebe48173d96bd690"} Jul 13 21:59:21.429231 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.429202 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cfc77bcff-76rjs" event={"ID":"d7ee462f-da9d-40ec-a8b7-b1f49e79174c","Type":"ContainerStarted","Data":"772233439bed21aafb1f565172234d4e0f91e38667b6b55106815254c788753c"} Jul 13 21:59:21.431572 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.431531 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-78th4" event={"ID":"49ff524c-8e50-4f1e-adf4-778791a3191e","Type":"ContainerStarted","Data":"0c2748d2872f9012febceec63fed91ede750d0ff25e327c9281791270fadbe7c"} Jul 13 21:59:21.436730 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.436666 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b445969f5-p5986" event={"ID":"c749ca98-5b06-4ff7-9282-3c72a1d8b036","Type":"ContainerStarted","Data":"2dcc0be614b10d832b3cbe7ac0345ea679586c2106fe43a51cfb8342e7f916b6"} Jul 13 21:59:21.437819 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.437774 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9dkjb" podStartSLOduration=35.630257587 podStartE2EDuration="39.437757539s" totalImagesPullingTime="3.807499952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:21.435988459 +0000 UTC m=+74.093193228" watchObservedRunningTime="2026-07-13 21:59:21.437757539 +0000 UTC m=+74.094962308" Jul 13 21:59:21.440472 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.440083 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qv2xs" event={"ID":"2b1adc79-d0de-44ab-b9c5-747c579b20cb","Type":"ContainerStarted","Data":"3fd2ecf4a3dbe162a9f2e0238f09d3875c7500cdaf84c5cd26a512291e670dc7"} Jul 13 21:59:21.457413 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.457301 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-stpjr" podStartSLOduration=56.887969205 podStartE2EDuration="1m0.457282966s" totalImagesPullingTime="3.569313761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:21.45666233 +0000 UTC m=+74.113867104" watchObservedRunningTime="2026-07-13 21:59:21.457282966 +0000 UTC m=+74.114487735" Jul 13 21:59:21.539352 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.539288 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qv2xs" podStartSLOduration=67.433819474 podStartE2EDuration="1m13.539266188s" totalImagesPullingTime="6.105446714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:21.471158993 +0000 UTC m=+74.128363765" watchObservedRunningTime="2026-07-13 21:59:21.539266188 +0000 UTC m=+74.196470957" Jul 13 21:59:21.541226 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:21.541157 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pqh9"] Jul 13 21:59:21.546520 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:59:21.546495 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557b4511_3dad_4e13_92cb_0d5c4eef6af5.slice/crio-9173f6f46514dceba279f381260d6b5aa11dbc95a18574f73abec041b1f5114f WatchSource:0}: Error finding container 9173f6f46514dceba279f381260d6b5aa11dbc95a18574f73abec041b1f5114f: Status 404 returned error can't find the container with id 9173f6f46514dceba279f381260d6b5aa11dbc95a18574f73abec041b1f5114f Jul 13 21:59:22.463302 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:22.462261 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pqh9" event={"ID":"557b4511-3dad-4e13-92cb-0d5c4eef6af5","Type":"ContainerStarted","Data":"9173f6f46514dceba279f381260d6b5aa11dbc95a18574f73abec041b1f5114f"} Jul 13 21:59:30.517220 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:30.517182 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c788876d8-ww7qf" event={"ID":"259a215e-706d-4578-99cc-7f236307874c","Type":"ContainerStarted","Data":"e312a94f85256762429d231dbc83477eddd4d2dab12b78c0f87c402a1b620961"} Jul 13 21:59:30.517719 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:30.517369 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c788876d8-ww7qf" Jul 13 21:59:30.518956 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:30.518931 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749684677d-rdjxl" event={"ID":"c541c830-4b9b-43a1-8351-222c81fa2368","Type":"ContainerStarted","Data":"caae6353942e24e2221967d88f5e142e98339e87ca8a8183363e63468d75ee27"} Jul 13 21:59:30.519086 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:30.519049 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c788876d8-ww7qf" Jul 13 21:59:30.521002 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:30.520889 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" event={"ID":"cf4c85ba-e813-4696-a51a-a6b8a353a294","Type":"ContainerStarted","Data":"6c41346bf8a884d3da2aae06c3c69e45fd21ab4175029883659850fbd2c91363"} Jul 13 21:59:30.521002 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:30.520925 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" event={"ID":"cf4c85ba-e813-4696-a51a-a6b8a353a294","Type":"ContainerStarted","Data":"99de6a22693c5b68f6f81441b29a6c4d2949083e40e1e107346bd5cf0f2cdf74"} Jul 13 21:59:30.521002 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:30.520945 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" event={"ID":"cf4c85ba-e813-4696-a51a-a6b8a353a294","Type":"ContainerStarted","Data":"c4f7ca25df5d57550445adbe2256a53790935cf53c4f1e1bb7f5d118e00b428d"} Jul 13 21:59:30.522434 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:30.522414 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cfc77bcff-76rjs" event={"ID":"d7ee462f-da9d-40ec-a8b7-b1f49e79174c","Type":"ContainerStarted","Data":"e130711f841752c0def19bdc8734822be7a1dca64391dbb661fd9e039b37cc66"} Jul 13 21:59:30.523895 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:30.523877 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b445969f5-p5986" event={"ID":"c749ca98-5b06-4ff7-9282-3c72a1d8b036","Type":"ContainerStarted","Data":"379d6265fdb8d7a3105468506a8e25405174852ed9624aad103ce5d20759c502"} Jul 13 21:59:30.525102 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:30.525083 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pqh9" event={"ID":"557b4511-3dad-4e13-92cb-0d5c4eef6af5","Type":"ContainerStarted","Data":"8fdc0090840a5548d5370e519f79d89a291196f053df7587ff2c94be5c16808d"} Jul 13 21:59:30.525324 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:30.525249 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pqh9" Jul 13 21:59:30.529783 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:30.529762 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pqh9" Jul 13 21:59:30.535555 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:30.535503 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c788876d8-ww7qf" podStartSLOduration=2.145873303 podStartE2EDuration="11.535490486s" totalImagesPullingTime="9.389617183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:30.534768994 +0000 UTC m=+83.191973764" watchObservedRunningTime="2026-07-13 21:59:30.535490486 +0000 UTC m=+83.192695293" Jul 13 21:59:30.551726 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:30.551677 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cfc77bcff-76rjs" podStartSLOduration=2.185504002 podStartE2EDuration="11.551660589s" totalImagesPullingTime="9.366156587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:30.551229621 +0000 UTC m=+83.208434390" watchObservedRunningTime="2026-07-13 21:59:30.551660589 +0000 UTC m=+83.208865358" Jul 13 21:59:30.569390 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:30.569343 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pqh9" podStartSLOduration=2.063530331 podStartE2EDuration="10.569326954s" totalImagesPullingTime="8.505796623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:59:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:30.567803588 +0000 UTC m=+83.225008356" watchObservedRunningTime="2026-07-13 21:59:30.569326954 +0000 UTC m=+83.226531703" Jul 13 21:59:30.603451 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:30.603408 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb794d48-pmrmv" podStartSLOduration=2.271951657 podStartE2EDuration="11.603390955s" totalImagesPullingTime="9.331439298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:30.603282168 +0000 UTC m=+83.260486937" watchObservedRunningTime="2026-07-13 21:59:30.603390955 +0000 UTC m=+83.260595726" Jul 13 21:59:30.623966 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:30.623919 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-749684677d-rdjxl" podStartSLOduration=2.251755442 podStartE2EDuration="11.623901843s" totalImagesPullingTime="9.372146401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:30.62313408 +0000 UTC m=+83.280338848" watchObservedRunningTime="2026-07-13 21:59:30.623901843 +0000 UTC m=+83.281106615" Jul 13 21:59:30.640515 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:30.640395 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b445969f5-p5986" podStartSLOduration=2.4202466830000002 podStartE2EDuration="11.640379729s" totalImagesPullingTime="9.220133046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:30.63961136 +0000 UTC m=+83.296816129" watchObservedRunningTime="2026-07-13 21:59:30.640379729 +0000 UTC m=+83.297584513" Jul 13 21:59:31.466548 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:31.466516 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9dkjb" Jul 13 21:59:34.790951 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:34.790899 2569 patch_prober.go:28] interesting pod/image-registry-7f784dd4c9-l4qbc container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 13 21:59:34.791465 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:34.790976 2569 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" podUID="d6cac02d-f5da-4569-894c-0328af666e63" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 21:59:35.459058 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.458991 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ptsjg"] Jul 13 21:59:35.477131 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.476808 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.482862 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.482836 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 13 21:59:35.482975 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.482888 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 13 21:59:35.482975 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.482903 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 13 21:59:35.483974 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.483927 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 13 21:59:35.484204 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.484119 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lqdmm\"" Jul 13 21:59:35.538448 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.538158 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.538448 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.538218 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-root\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.538448 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.538290 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-node-exporter-textfile\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.538448 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.538317 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-node-exporter-tls\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.538448 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.538362 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-metrics-client-ca\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.538448 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.538384 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-node-exporter-accelerators-collector-config\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.538448 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.538424 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-sys\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.538890 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.538502 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-node-exporter-wtmp\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.538890 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.538570 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6c7x\" (UniqueName: \"kubernetes.io/projected/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-kube-api-access-n6c7x\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.640344 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.639736 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-sys\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.640344 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.639788 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-node-exporter-wtmp\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.640344 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.639811 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-sys\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.640344 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.639819 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6c7x\" (UniqueName: \"kubernetes.io/projected/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-kube-api-access-n6c7x\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.640344 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.639895 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.640344 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.639925 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-root\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.640344 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.639983 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-node-exporter-textfile\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.640344 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.639990 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-node-exporter-wtmp\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.640344 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.640014 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-node-exporter-tls\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.640344 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.640036 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-root\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.640344 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.640053 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-metrics-client-ca\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.640344 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.640077 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-node-exporter-accelerators-collector-config\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.640344 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:59:35.640153 2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 13 21:59:35.640344 ip-10-0-137-183 kubenswrapper[2569]: E0713 21:59:35.640209 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-node-exporter-tls podName:e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa nodeName:}" failed. No retries permitted until 2026-07-13 21:59:36.140190137 +0000 UTC m=+88.797394891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-node-exporter-tls") pod "node-exporter-ptsjg" (UID: "e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa") : secret "node-exporter-tls" not found Jul 13 21:59:35.641150 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.640851 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-node-exporter-textfile\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.641150 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.640915 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-node-exporter-accelerators-collector-config\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.641258 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.641231 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-metrics-client-ca\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.642874 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.642816 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:35.653866 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:35.653814 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6c7x\" (UniqueName: \"kubernetes.io/projected/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-kube-api-access-n6c7x\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:36.144780 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:36.144739 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-node-exporter-tls\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:36.147633 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:36.147611 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa-node-exporter-tls\") pod \"node-exporter-ptsjg\" (UID: \"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa\") " pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:36.394267 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:36.394225 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ptsjg" Jul 13 21:59:38.369208 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:38.369178 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 21:59:39.756420 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:39.756363 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:39.756913 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:39.756703 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:39.761936 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:39.761914 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:40.259834 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:40.259791 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:40.260097 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:40.259860 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:40.266054 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:40.265985 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b445969f5-p5986"] Jul 13 21:59:40.266298 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:40.266254 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:40.586496 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:40.586424 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b445969f5-p5986" Jul 13 21:59:40.586667 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:40.586497 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 21:59:41.500907 ip-10-0-137-183 kubenswrapper[2569]: W0713 21:59:41.500877 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode595f8f1_bc53_4bf3_9ba8_815c3d53ecfa.slice/crio-2165ebfc16e21e0f94320f43866223176e463c66f4e56c6513f39f01a559c6d0 WatchSource:0}: Error finding container 2165ebfc16e21e0f94320f43866223176e463c66f4e56c6513f39f01a559c6d0: Status 404 returned error can't find the container with id 2165ebfc16e21e0f94320f43866223176e463c66f4e56c6513f39f01a559c6d0 Jul 13 21:59:41.585509 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:41.585472 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ptsjg" event={"ID":"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa","Type":"ContainerStarted","Data":"2165ebfc16e21e0f94320f43866223176e463c66f4e56c6513f39f01a559c6d0"} Jul 13 21:59:42.592368 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:42.592331 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-78th4" event={"ID":"49ff524c-8e50-4f1e-adf4-778791a3191e","Type":"ContainerStarted","Data":"f785071a833dc9b62fe19777a93a5662d0e22ab86ea98571bb08c6749c96b69e"} Jul 13 21:59:42.592911 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:42.592889 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-78th4" Jul 13 21:59:42.612643 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:42.612618 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-78th4" Jul 13 21:59:42.629273 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:42.629179 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-78th4" podStartSLOduration=2.247436234 podStartE2EDuration="23.629164566s" totalImagesPullingTime="21.381728332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:42.610301634 +0000 UTC m=+95.267506401" watchObservedRunningTime="2026-07-13 21:59:42.629164566 +0000 UTC m=+95.286369334" Jul 13 21:59:43.599949 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:43.599909 2569 generic.go:356] "Generic (PLEG): container finished" podID="e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa" containerID="24068baf2bb9ecebf37e12aad7789163e127e868ddea554a91329da5ed4b9611" exitCode=0 Jul 13 21:59:43.600528 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:43.600010 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ptsjg" event={"ID":"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa","Type":"ContainerDied","Data":"24068baf2bb9ecebf37e12aad7789163e127e868ddea554a91329da5ed4b9611"} Jul 13 21:59:44.608234 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:44.608188 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ptsjg" event={"ID":"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa","Type":"ContainerStarted","Data":"b092f594e3351c93628449c2a187e1fdfd1268f05474c13138721807948655de"} Jul 13 21:59:44.608687 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:44.608260 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ptsjg" event={"ID":"e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa","Type":"ContainerStarted","Data":"aad29342340b352be0cef0299d7236e51c9e196eb83cb2938af7acf21e0684cf"} Jul 13 21:59:44.628460 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:44.628408 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ptsjg" podStartSLOduration=8.114344183 podStartE2EDuration="9.628393088s" totalImagesPullingTime="1.40577764s" totalInitContainerRuntime="108.271265ms" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:44.627360221 +0000 UTC m=+97.284564989" watchObservedRunningTime="2026-07-13 21:59:44.628393088 +0000 UTC m=+97.285597856" Jul 13 21:59:45.334116 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:45.334080 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qmrjq" Jul 13 21:59:48.202099 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:48.202065 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f784dd4c9-l4qbc"] Jul 13 21:59:58.337590 ip-10-0-137-183 kubenswrapper[2569]: I0713 21:59:58.337558 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cfc77bcff-76rjs"] Jul 13 22:00:01.686485 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:01.686455 2569 generic.go:356] "Generic (PLEG): container finished" podID="80fea354-ee7a-49af-8201-0bc32959c2d0" containerID="6fcbe817cd703595c64cc2c36c818c7b5758e506203fbe018fdde07d311c8e24" exitCode=0 Jul 13 22:00:01.686902 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:01.686525 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g4ds9" event={"ID":"80fea354-ee7a-49af-8201-0bc32959c2d0","Type":"ContainerDied","Data":"6fcbe817cd703595c64cc2c36c818c7b5758e506203fbe018fdde07d311c8e24"} Jul 13 22:00:01.686902 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:01.686857 2569 scope.go:117] "RemoveContainer" containerID="6fcbe817cd703595c64cc2c36c818c7b5758e506203fbe018fdde07d311c8e24" Jul 13 22:00:02.692740 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:02.692706 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g4ds9" event={"ID":"80fea354-ee7a-49af-8201-0bc32959c2d0","Type":"ContainerStarted","Data":"d58578d80d21489fe11f09ecf89a929c8b5e6a41cf1fc4bb4fd97e10ef12ad8a"} Jul 13 22:00:02.694152 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:02.694130 2569 generic.go:356] "Generic (PLEG): container finished" podID="8d45c661-160c-4a4e-a7b8-c6e055315ae1" containerID="0d321fefdc72203948692ab7f00a4ad983821df9d72b7a070715ce5e72639b80" exitCode=0 Jul 13 22:00:02.694269 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:02.694169 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zghwn" event={"ID":"8d45c661-160c-4a4e-a7b8-c6e055315ae1","Type":"ContainerDied","Data":"0d321fefdc72203948692ab7f00a4ad983821df9d72b7a070715ce5e72639b80"} Jul 13 22:00:02.694406 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:02.694394 2569 scope.go:117] "RemoveContainer" containerID="0d321fefdc72203948692ab7f00a4ad983821df9d72b7a070715ce5e72639b80" Jul 13 22:00:03.701202 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:03.701171 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zghwn" event={"ID":"8d45c661-160c-4a4e-a7b8-c6e055315ae1","Type":"ContainerStarted","Data":"6c82ccb1c89482e01c91cedaf36f186e7e13702cf2724b518698117cc5f10e37"} Jul 13 22:00:05.608464 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:05.608425 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b445969f5-p5986" podUID="c749ca98-5b06-4ff7-9282-3c72a1d8b036" containerName="console" containerID="cri-o://379d6265fdb8d7a3105468506a8e25405174852ed9624aad103ce5d20759c502" gracePeriod=15 Jul 13 22:00:05.890184 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:05.890122 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b445969f5-p5986" Jul 13 22:00:06.000590 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.000564 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-service-ca\") pod \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " Jul 13 22:00:06.000743 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.000626 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-oauth-serving-cert\") pod \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " Jul 13 22:00:06.000743 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.000654 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c749ca98-5b06-4ff7-9282-3c72a1d8b036-console-oauth-config\") pod \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " Jul 13 22:00:06.000743 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.000694 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c749ca98-5b06-4ff7-9282-3c72a1d8b036-console-serving-cert\") pod \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " Jul 13 22:00:06.000743 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.000719 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-console-config\") pod \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " Jul 13 22:00:06.000946 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.000753 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpfw8\" (UniqueName: \"kubernetes.io/projected/c749ca98-5b06-4ff7-9282-3c72a1d8b036-kube-api-access-tpfw8\") pod \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " Jul 13 22:00:06.000946 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.000782 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-trusted-ca-bundle\") pod \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\" (UID: \"c749ca98-5b06-4ff7-9282-3c72a1d8b036\") " Jul 13 22:00:06.001039 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.000899 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-service-ca" (OuterVolumeSpecName: "service-ca") pod "c749ca98-5b06-4ff7-9282-3c72a1d8b036" (UID: "c749ca98-5b06-4ff7-9282-3c72a1d8b036"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:00:06.001130 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.001106 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c749ca98-5b06-4ff7-9282-3c72a1d8b036" (UID: "c749ca98-5b06-4ff7-9282-3c72a1d8b036"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:00:06.001191 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.001116 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-service-ca\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:06.001191 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.001146 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-console-config" (OuterVolumeSpecName: "console-config") pod "c749ca98-5b06-4ff7-9282-3c72a1d8b036" (UID: "c749ca98-5b06-4ff7-9282-3c72a1d8b036"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:00:06.001280 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.001210 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c749ca98-5b06-4ff7-9282-3c72a1d8b036" (UID: "c749ca98-5b06-4ff7-9282-3c72a1d8b036"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:00:06.003100 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.003076 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c749ca98-5b06-4ff7-9282-3c72a1d8b036-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c749ca98-5b06-4ff7-9282-3c72a1d8b036" (UID: "c749ca98-5b06-4ff7-9282-3c72a1d8b036"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:00:06.003222 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.003189 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c749ca98-5b06-4ff7-9282-3c72a1d8b036-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c749ca98-5b06-4ff7-9282-3c72a1d8b036" (UID: "c749ca98-5b06-4ff7-9282-3c72a1d8b036"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:00:06.003222 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.003189 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c749ca98-5b06-4ff7-9282-3c72a1d8b036-kube-api-access-tpfw8" (OuterVolumeSpecName: "kube-api-access-tpfw8") pod "c749ca98-5b06-4ff7-9282-3c72a1d8b036" (UID: "c749ca98-5b06-4ff7-9282-3c72a1d8b036"). InnerVolumeSpecName "kube-api-access-tpfw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:00:06.101496 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.101474 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-oauth-serving-cert\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:06.101496 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.101496 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c749ca98-5b06-4ff7-9282-3c72a1d8b036-console-oauth-config\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:06.101650 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.101505 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c749ca98-5b06-4ff7-9282-3c72a1d8b036-console-serving-cert\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:06.101650 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.101514 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-console-config\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:06.101650 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.101523 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpfw8\" (UniqueName: \"kubernetes.io/projected/c749ca98-5b06-4ff7-9282-3c72a1d8b036-kube-api-access-tpfw8\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:06.101650 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.101557 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c749ca98-5b06-4ff7-9282-3c72a1d8b036-trusted-ca-bundle\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:06.716873 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.716835 2569 generic.go:356] "Generic (PLEG): container finished" podID="c749ca98-5b06-4ff7-9282-3c72a1d8b036" containerID="379d6265fdb8d7a3105468506a8e25405174852ed9624aad103ce5d20759c502" exitCode=0 Jul 13 22:00:06.717305 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.716887 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b445969f5-p5986" Jul 13 22:00:06.717305 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.716919 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b445969f5-p5986" event={"ID":"c749ca98-5b06-4ff7-9282-3c72a1d8b036","Type":"ContainerDied","Data":"379d6265fdb8d7a3105468506a8e25405174852ed9624aad103ce5d20759c502"} Jul 13 22:00:06.717305 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.716966 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b445969f5-p5986" event={"ID":"c749ca98-5b06-4ff7-9282-3c72a1d8b036","Type":"ContainerDied","Data":"2dcc0be614b10d832b3cbe7ac0345ea679586c2106fe43a51cfb8342e7f916b6"} Jul 13 22:00:06.717305 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.716985 2569 scope.go:117] "RemoveContainer" containerID="379d6265fdb8d7a3105468506a8e25405174852ed9624aad103ce5d20759c502" Jul 13 22:00:06.726866 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.726843 2569 scope.go:117] "RemoveContainer" containerID="379d6265fdb8d7a3105468506a8e25405174852ed9624aad103ce5d20759c502" Jul 13 22:00:06.727201 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:00:06.727176 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379d6265fdb8d7a3105468506a8e25405174852ed9624aad103ce5d20759c502\": container with ID starting with 379d6265fdb8d7a3105468506a8e25405174852ed9624aad103ce5d20759c502 not found: ID does not exist" containerID="379d6265fdb8d7a3105468506a8e25405174852ed9624aad103ce5d20759c502" Jul 13 22:00:06.727278 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.727213 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379d6265fdb8d7a3105468506a8e25405174852ed9624aad103ce5d20759c502"} err="failed to get container status \"379d6265fdb8d7a3105468506a8e25405174852ed9624aad103ce5d20759c502\": rpc error: code = NotFound desc = could not find container \"379d6265fdb8d7a3105468506a8e25405174852ed9624aad103ce5d20759c502\": container with ID starting with 379d6265fdb8d7a3105468506a8e25405174852ed9624aad103ce5d20759c502 not found: ID does not exist" Jul 13 22:00:06.737645 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.737624 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b445969f5-p5986"] Jul 13 22:00:06.741424 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:06.741404 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b445969f5-p5986"] Jul 13 22:00:07.931469 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:07.931437 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c749ca98-5b06-4ff7-9282-3c72a1d8b036" path="/var/lib/kubelet/pods/c749ca98-5b06-4ff7-9282-3c72a1d8b036/volumes" Jul 13 22:00:13.227870 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.227807 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" podUID="d6cac02d-f5da-4569-894c-0328af666e63" containerName="registry" containerID="cri-o://15bbe1b22f1956c1b56b5141300cf23dee814e873f26a3375e343b0e437deb69" gracePeriod=30 Jul 13 22:00:13.465439 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.465417 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 22:00:13.560277 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.560215 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6cac02d-f5da-4569-894c-0328af666e63-installation-pull-secrets\") pod \"d6cac02d-f5da-4569-894c-0328af666e63\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " Jul 13 22:00:13.560277 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.560257 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls\") pod \"d6cac02d-f5da-4569-894c-0328af666e63\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " Jul 13 22:00:13.560480 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.560297 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6cac02d-f5da-4569-894c-0328af666e63-image-registry-private-configuration\") pod \"d6cac02d-f5da-4569-894c-0328af666e63\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " Jul 13 22:00:13.560480 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.560353 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-bound-sa-token\") pod \"d6cac02d-f5da-4569-894c-0328af666e63\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " Jul 13 22:00:13.560480 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.560380 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6cac02d-f5da-4569-894c-0328af666e63-trusted-ca\") pod \"d6cac02d-f5da-4569-894c-0328af666e63\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " Jul 13 22:00:13.560480 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.560414 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6cac02d-f5da-4569-894c-0328af666e63-registry-certificates\") pod \"d6cac02d-f5da-4569-894c-0328af666e63\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " Jul 13 22:00:13.560480 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.560468 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6cac02d-f5da-4569-894c-0328af666e63-ca-trust-extracted\") pod \"d6cac02d-f5da-4569-894c-0328af666e63\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " Jul 13 22:00:13.560769 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.560527 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4c27\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-kube-api-access-b4c27\") pod \"d6cac02d-f5da-4569-894c-0328af666e63\" (UID: \"d6cac02d-f5da-4569-894c-0328af666e63\") " Jul 13 22:00:13.560955 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.560929 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6cac02d-f5da-4569-894c-0328af666e63-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d6cac02d-f5da-4569-894c-0328af666e63" (UID: "d6cac02d-f5da-4569-894c-0328af666e63"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:00:13.561463 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.561434 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6cac02d-f5da-4569-894c-0328af666e63-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d6cac02d-f5da-4569-894c-0328af666e63" (UID: "d6cac02d-f5da-4569-894c-0328af666e63"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:00:13.562959 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.562886 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d6cac02d-f5da-4569-894c-0328af666e63" (UID: "d6cac02d-f5da-4569-894c-0328af666e63"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:00:13.562959 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.562892 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6cac02d-f5da-4569-894c-0328af666e63-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d6cac02d-f5da-4569-894c-0328af666e63" (UID: "d6cac02d-f5da-4569-894c-0328af666e63"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:00:13.563118 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.562974 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6cac02d-f5da-4569-894c-0328af666e63-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d6cac02d-f5da-4569-894c-0328af666e63" (UID: "d6cac02d-f5da-4569-894c-0328af666e63"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:00:13.563118 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.563046 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d6cac02d-f5da-4569-894c-0328af666e63" (UID: "d6cac02d-f5da-4569-894c-0328af666e63"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:00:13.563214 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.563126 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-kube-api-access-b4c27" (OuterVolumeSpecName: "kube-api-access-b4c27") pod "d6cac02d-f5da-4569-894c-0328af666e63" (UID: "d6cac02d-f5da-4569-894c-0328af666e63"). InnerVolumeSpecName "kube-api-access-b4c27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:00:13.572138 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.572111 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cac02d-f5da-4569-894c-0328af666e63-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d6cac02d-f5da-4569-894c-0328af666e63" (UID: "d6cac02d-f5da-4569-894c-0328af666e63"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:00:13.661095 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.661072 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4c27\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-kube-api-access-b4c27\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:13.661095 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.661093 2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6cac02d-f5da-4569-894c-0328af666e63-installation-pull-secrets\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:13.661206 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.661103 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-registry-tls\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:13.661206 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.661114 2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6cac02d-f5da-4569-894c-0328af666e63-image-registry-private-configuration\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:13.661206 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.661123 2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6cac02d-f5da-4569-894c-0328af666e63-bound-sa-token\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:13.661206 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.661131 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6cac02d-f5da-4569-894c-0328af666e63-trusted-ca\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:13.661206 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.661139 2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6cac02d-f5da-4569-894c-0328af666e63-registry-certificates\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:13.661206 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.661147 2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6cac02d-f5da-4569-894c-0328af666e63-ca-trust-extracted\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:13.747851 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.747821 2569 generic.go:356] "Generic (PLEG): container finished" podID="d6cac02d-f5da-4569-894c-0328af666e63" containerID="15bbe1b22f1956c1b56b5141300cf23dee814e873f26a3375e343b0e437deb69" exitCode=0 Jul 13 22:00:13.747956 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.747863 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" Jul 13 22:00:13.747956 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.747872 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" event={"ID":"d6cac02d-f5da-4569-894c-0328af666e63","Type":"ContainerDied","Data":"15bbe1b22f1956c1b56b5141300cf23dee814e873f26a3375e343b0e437deb69"} Jul 13 22:00:13.747956 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.747914 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f784dd4c9-l4qbc" event={"ID":"d6cac02d-f5da-4569-894c-0328af666e63","Type":"ContainerDied","Data":"647ab0ae1676010b639919fe599e6a42a69c56ca763df63030021d0f37a28bc4"} Jul 13 22:00:13.747956 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.747938 2569 scope.go:117] "RemoveContainer" containerID="15bbe1b22f1956c1b56b5141300cf23dee814e873f26a3375e343b0e437deb69" Jul 13 22:00:13.756280 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.756263 2569 scope.go:117] "RemoveContainer" containerID="15bbe1b22f1956c1b56b5141300cf23dee814e873f26a3375e343b0e437deb69" Jul 13 22:00:13.756546 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:00:13.756513 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15bbe1b22f1956c1b56b5141300cf23dee814e873f26a3375e343b0e437deb69\": container with ID starting with 15bbe1b22f1956c1b56b5141300cf23dee814e873f26a3375e343b0e437deb69 not found: ID does not exist" containerID="15bbe1b22f1956c1b56b5141300cf23dee814e873f26a3375e343b0e437deb69" Jul 13 22:00:13.756614 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.756560 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15bbe1b22f1956c1b56b5141300cf23dee814e873f26a3375e343b0e437deb69"} err="failed to get container status \"15bbe1b22f1956c1b56b5141300cf23dee814e873f26a3375e343b0e437deb69\": rpc error: code = NotFound desc = could not find container \"15bbe1b22f1956c1b56b5141300cf23dee814e873f26a3375e343b0e437deb69\": container with ID starting with 15bbe1b22f1956c1b56b5141300cf23dee814e873f26a3375e343b0e437deb69 not found: ID does not exist" Jul 13 22:00:13.768223 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.768200 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f784dd4c9-l4qbc"] Jul 13 22:00:13.771965 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.771947 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f784dd4c9-l4qbc"] Jul 13 22:00:13.930135 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:13.930060 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6cac02d-f5da-4569-894c-0328af666e63" path="/var/lib/kubelet/pods/d6cac02d-f5da-4569-894c-0328af666e63/volumes" Jul 13 22:00:21.788550 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:21.788502 2569 generic.go:356] "Generic (PLEG): container finished" podID="3a59df9a-555c-4676-9b35-d1f8b2752e82" containerID="3dc61d28ccc0e5470607fecd448f21e97352685c123baa876b95bdca1f37582c" exitCode=0 Jul 13 22:00:21.788942 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:21.788573 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" event={"ID":"3a59df9a-555c-4676-9b35-d1f8b2752e82","Type":"ContainerDied","Data":"3dc61d28ccc0e5470607fecd448f21e97352685c123baa876b95bdca1f37582c"} Jul 13 22:00:21.788942 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:21.788907 2569 scope.go:117] "RemoveContainer" containerID="3dc61d28ccc0e5470607fecd448f21e97352685c123baa876b95bdca1f37582c" Jul 13 22:00:22.793907 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:22.793873 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-85gwn" event={"ID":"3a59df9a-555c-4676-9b35-d1f8b2752e82","Type":"ContainerStarted","Data":"16c59e4ff50c0b6b600d8f112415f045d2a3b714973416b9bcc6125e26e4c8fa"} Jul 13 22:00:23.360197 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.360161 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cfc77bcff-76rjs" podUID="d7ee462f-da9d-40ec-a8b7-b1f49e79174c" containerName="console" containerID="cri-o://e130711f841752c0def19bdc8734822be7a1dca64391dbb661fd9e039b37cc66" gracePeriod=15 Jul 13 22:00:23.634853 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.634772 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 22:00:23.736056 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.736025 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-oauth-serving-cert\") pod \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " Jul 13 22:00:23.736056 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.736070 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-console-oauth-config\") pod \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " Jul 13 22:00:23.736247 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.736095 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-console-config\") pod \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " Jul 13 22:00:23.736247 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.736130 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f6w8\" (UniqueName: \"kubernetes.io/projected/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-kube-api-access-6f6w8\") pod \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " Jul 13 22:00:23.736247 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.736174 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-service-ca\") pod \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " Jul 13 22:00:23.736247 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.736217 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-console-serving-cert\") pod \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\" (UID: \"d7ee462f-da9d-40ec-a8b7-b1f49e79174c\") " Jul 13 22:00:23.736525 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.736498 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-console-config" (OuterVolumeSpecName: "console-config") pod "d7ee462f-da9d-40ec-a8b7-b1f49e79174c" (UID: "d7ee462f-da9d-40ec-a8b7-b1f49e79174c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:00:23.736696 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.736668 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-service-ca" (OuterVolumeSpecName: "service-ca") pod "d7ee462f-da9d-40ec-a8b7-b1f49e79174c" (UID: "d7ee462f-da9d-40ec-a8b7-b1f49e79174c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:00:23.736778 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.736725 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d7ee462f-da9d-40ec-a8b7-b1f49e79174c" (UID: "d7ee462f-da9d-40ec-a8b7-b1f49e79174c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:00:23.738311 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.738276 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d7ee462f-da9d-40ec-a8b7-b1f49e79174c" (UID: "d7ee462f-da9d-40ec-a8b7-b1f49e79174c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:00:23.738441 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.738366 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d7ee462f-da9d-40ec-a8b7-b1f49e79174c" (UID: "d7ee462f-da9d-40ec-a8b7-b1f49e79174c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:00:23.738441 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.738394 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-kube-api-access-6f6w8" (OuterVolumeSpecName: "kube-api-access-6f6w8") pod "d7ee462f-da9d-40ec-a8b7-b1f49e79174c" (UID: "d7ee462f-da9d-40ec-a8b7-b1f49e79174c"). InnerVolumeSpecName "kube-api-access-6f6w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:00:23.799031 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.798992 2569 generic.go:356] "Generic (PLEG): container finished" podID="d7ee462f-da9d-40ec-a8b7-b1f49e79174c" containerID="e130711f841752c0def19bdc8734822be7a1dca64391dbb661fd9e039b37cc66" exitCode=0 Jul 13 22:00:23.799382 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.799038 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cfc77bcff-76rjs" Jul 13 22:00:23.799382 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.799077 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cfc77bcff-76rjs" event={"ID":"d7ee462f-da9d-40ec-a8b7-b1f49e79174c","Type":"ContainerDied","Data":"e130711f841752c0def19bdc8734822be7a1dca64391dbb661fd9e039b37cc66"} Jul 13 22:00:23.799382 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.799130 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cfc77bcff-76rjs" event={"ID":"d7ee462f-da9d-40ec-a8b7-b1f49e79174c","Type":"ContainerDied","Data":"772233439bed21aafb1f565172234d4e0f91e38667b6b55106815254c788753c"} Jul 13 22:00:23.799382 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.799151 2569 scope.go:117] "RemoveContainer" containerID="e130711f841752c0def19bdc8734822be7a1dca64391dbb661fd9e039b37cc66" Jul 13 22:00:23.807930 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.807902 2569 scope.go:117] "RemoveContainer" containerID="e130711f841752c0def19bdc8734822be7a1dca64391dbb661fd9e039b37cc66" Jul 13 22:00:23.809868 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:00:23.808247 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e130711f841752c0def19bdc8734822be7a1dca64391dbb661fd9e039b37cc66\": container with ID starting with e130711f841752c0def19bdc8734822be7a1dca64391dbb661fd9e039b37cc66 not found: ID does not exist" containerID="e130711f841752c0def19bdc8734822be7a1dca64391dbb661fd9e039b37cc66" Jul 13 22:00:23.809868 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.808287 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e130711f841752c0def19bdc8734822be7a1dca64391dbb661fd9e039b37cc66"} err="failed to get container status \"e130711f841752c0def19bdc8734822be7a1dca64391dbb661fd9e039b37cc66\": rpc error: code = NotFound desc = could not find container \"e130711f841752c0def19bdc8734822be7a1dca64391dbb661fd9e039b37cc66\": container with ID starting with e130711f841752c0def19bdc8734822be7a1dca64391dbb661fd9e039b37cc66 not found: ID does not exist" Jul 13 22:00:23.823509 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.823485 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cfc77bcff-76rjs"] Jul 13 22:00:23.825914 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.825890 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cfc77bcff-76rjs"] Jul 13 22:00:23.837415 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.837396 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-console-serving-cert\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:23.837516 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.837418 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-oauth-serving-cert\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:23.837516 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.837432 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-console-oauth-config\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:23.837516 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.837445 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-console-config\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:23.837516 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.837458 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f6w8\" (UniqueName: \"kubernetes.io/projected/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-kube-api-access-6f6w8\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:23.837516 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.837470 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ee462f-da9d-40ec-a8b7-b1f49e79174c-service-ca\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:00:23.929656 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:00:23.929581 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7ee462f-da9d-40ec-a8b7-b1f49e79174c" path="/var/lib/kubelet/pods/d7ee462f-da9d-40ec-a8b7-b1f49e79174c/volumes" Jul 13 22:02:59.564653 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.564620 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms"] Jul 13 22:02:59.565047 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.564926 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6cac02d-f5da-4569-894c-0328af666e63" containerName="registry" Jul 13 22:02:59.565047 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.564938 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6cac02d-f5da-4569-894c-0328af666e63" containerName="registry" Jul 13 22:02:59.565047 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.564951 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c749ca98-5b06-4ff7-9282-3c72a1d8b036" containerName="console" Jul 13 22:02:59.565047 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.564957 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c749ca98-5b06-4ff7-9282-3c72a1d8b036" containerName="console" Jul 13 22:02:59.565047 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.564970 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7ee462f-da9d-40ec-a8b7-b1f49e79174c" containerName="console" Jul 13 22:02:59.565047 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.564976 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7ee462f-da9d-40ec-a8b7-b1f49e79174c" containerName="console" Jul 13 22:02:59.565047 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.565027 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c749ca98-5b06-4ff7-9282-3c72a1d8b036" containerName="console" Jul 13 22:02:59.565047 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.565036 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7ee462f-da9d-40ec-a8b7-b1f49e79174c" containerName="console" Jul 13 22:02:59.565047 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.565045 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6cac02d-f5da-4569-894c-0328af666e63" containerName="registry" Jul 13 22:02:59.567242 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.567224 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" Jul 13 22:02:59.570569 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.570546 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 13 22:02:59.571458 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.571439 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 13 22:02:59.571849 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.571834 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 13 22:02:59.572303 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.572280 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8h4rb\"" Jul 13 22:02:59.572646 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.572621 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 13 22:02:59.585888 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.585865 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms"] Jul 13 22:02:59.635479 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.635450 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d29f9e2e-e085-4e61-a06f-df808c42f9d1-manifests\") pod \"opendatahub-operator-controller-manager-787f66975d-44mms\" (UID: \"d29f9e2e-e085-4e61-a06f-df808c42f9d1\") " pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" Jul 13 22:02:59.635479 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.635481 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d29f9e2e-e085-4e61-a06f-df808c42f9d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-787f66975d-44mms\" (UID: \"d29f9e2e-e085-4e61-a06f-df808c42f9d1\") " pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" Jul 13 22:02:59.635650 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.635502 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klwxv\" (UniqueName: \"kubernetes.io/projected/d29f9e2e-e085-4e61-a06f-df808c42f9d1-kube-api-access-klwxv\") pod \"opendatahub-operator-controller-manager-787f66975d-44mms\" (UID: \"d29f9e2e-e085-4e61-a06f-df808c42f9d1\") " pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" Jul 13 22:02:59.635650 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.635522 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d29f9e2e-e085-4e61-a06f-df808c42f9d1-tmp\") pod \"opendatahub-operator-controller-manager-787f66975d-44mms\" (UID: \"d29f9e2e-e085-4e61-a06f-df808c42f9d1\") " pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" Jul 13 22:02:59.635650 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.635589 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d29f9e2e-e085-4e61-a06f-df808c42f9d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-787f66975d-44mms\" (UID: \"d29f9e2e-e085-4e61-a06f-df808c42f9d1\") " pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" Jul 13 22:02:59.736517 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.736488 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d29f9e2e-e085-4e61-a06f-df808c42f9d1-manifests\") pod \"opendatahub-operator-controller-manager-787f66975d-44mms\" (UID: \"d29f9e2e-e085-4e61-a06f-df808c42f9d1\") " pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" Jul 13 22:02:59.736635 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.736521 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d29f9e2e-e085-4e61-a06f-df808c42f9d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-787f66975d-44mms\" (UID: \"d29f9e2e-e085-4e61-a06f-df808c42f9d1\") " pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" Jul 13 22:02:59.736635 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.736565 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klwxv\" (UniqueName: \"kubernetes.io/projected/d29f9e2e-e085-4e61-a06f-df808c42f9d1-kube-api-access-klwxv\") pod \"opendatahub-operator-controller-manager-787f66975d-44mms\" (UID: \"d29f9e2e-e085-4e61-a06f-df808c42f9d1\") " pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" Jul 13 22:02:59.736635 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.736595 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d29f9e2e-e085-4e61-a06f-df808c42f9d1-tmp\") pod \"opendatahub-operator-controller-manager-787f66975d-44mms\" (UID: \"d29f9e2e-e085-4e61-a06f-df808c42f9d1\") " pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" Jul 13 22:02:59.736635 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.736623 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d29f9e2e-e085-4e61-a06f-df808c42f9d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-787f66975d-44mms\" (UID: \"d29f9e2e-e085-4e61-a06f-df808c42f9d1\") " pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" Jul 13 22:02:59.736928 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.736906 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d29f9e2e-e085-4e61-a06f-df808c42f9d1-manifests\") pod \"opendatahub-operator-controller-manager-787f66975d-44mms\" (UID: \"d29f9e2e-e085-4e61-a06f-df808c42f9d1\") " pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" Jul 13 22:02:59.738971 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.738939 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d29f9e2e-e085-4e61-a06f-df808c42f9d1-tmp\") pod \"opendatahub-operator-controller-manager-787f66975d-44mms\" (UID: \"d29f9e2e-e085-4e61-a06f-df808c42f9d1\") " pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" Jul 13 22:02:59.739067 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.739031 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d29f9e2e-e085-4e61-a06f-df808c42f9d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-787f66975d-44mms\" (UID: \"d29f9e2e-e085-4e61-a06f-df808c42f9d1\") " pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" Jul 13 22:02:59.739067 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.739047 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d29f9e2e-e085-4e61-a06f-df808c42f9d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-787f66975d-44mms\" (UID: \"d29f9e2e-e085-4e61-a06f-df808c42f9d1\") " pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" Jul 13 22:02:59.745224 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.745202 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klwxv\" (UniqueName: \"kubernetes.io/projected/d29f9e2e-e085-4e61-a06f-df808c42f9d1-kube-api-access-klwxv\") pod \"opendatahub-operator-controller-manager-787f66975d-44mms\" (UID: \"d29f9e2e-e085-4e61-a06f-df808c42f9d1\") " pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" Jul 13 22:02:59.881593 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:02:59.881520 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" Jul 13 22:03:00.027260 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:00.027233 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms"] Jul 13 22:03:00.030313 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:03:00.030285 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29f9e2e_e085_4e61_a06f_df808c42f9d1.slice/crio-16c691b58bb501a47e8188af32355cb6d491203f7ccfc85c3d08ccc9c993f0e4 WatchSource:0}: Error finding container 16c691b58bb501a47e8188af32355cb6d491203f7ccfc85c3d08ccc9c993f0e4: Status 404 returned error can't find the container with id 16c691b58bb501a47e8188af32355cb6d491203f7ccfc85c3d08ccc9c993f0e4 Jul 13 22:03:00.408573 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:00.408527 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" event={"ID":"d29f9e2e-e085-4e61-a06f-df808c42f9d1","Type":"ContainerStarted","Data":"16c691b58bb501a47e8188af32355cb6d491203f7ccfc85c3d08ccc9c993f0e4"} Jul 13 22:03:03.424628 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:03.424590 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" event={"ID":"d29f9e2e-e085-4e61-a06f-df808c42f9d1","Type":"ContainerStarted","Data":"87e6b19863cda57c7cdd4992035ece7cfa5da1c738378e57d8c74d17674177d8"} Jul 13 22:03:04.430034 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:04.429939 2569 generic.go:356] "Generic (PLEG): container finished" podID="d29f9e2e-e085-4e61-a06f-df808c42f9d1" containerID="87e6b19863cda57c7cdd4992035ece7cfa5da1c738378e57d8c74d17674177d8" exitCode=0 Jul 13 22:03:04.430034 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:04.429998 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" event={"ID":"d29f9e2e-e085-4e61-a06f-df808c42f9d1","Type":"ContainerDied","Data":"87e6b19863cda57c7cdd4992035ece7cfa5da1c738378e57d8c74d17674177d8"} Jul 13 22:03:05.436262 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:05.436227 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" event={"ID":"d29f9e2e-e085-4e61-a06f-df808c42f9d1","Type":"ContainerStarted","Data":"3c2102bb8c3169e8b376e34437012576953809a1c2ca6df2d826b0dcda4e7097"} Jul 13 22:03:05.436657 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:05.436340 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" Jul 13 22:03:05.465274 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:05.463255 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" podStartSLOduration=2.221857435 podStartE2EDuration="6.463233763s" totalImagesPullingTime="3.213807673s" totalInitContainerRuntime="1.027568655s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:02:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:03:05.457885898 +0000 UTC m=+298.115090667" watchObservedRunningTime="2026-07-13 22:03:05.463233763 +0000 UTC m=+298.120438531" Jul 13 22:03:07.786379 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:07.786349 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 22:03:07.787147 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:07.787127 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 22:03:07.789722 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:07.789702 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 22:03:07.790458 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:07.790437 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 22:03:07.796806 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:07.796786 2569 kubelet.go:1648] "Image garbage collection succeeded" Jul 13 22:03:16.442746 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:16.442713 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-787f66975d-44mms" Jul 13 22:03:20.027097 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:20.027065 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-hfxp7"] Jul 13 22:03:20.029154 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:20.029135 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-hfxp7" Jul 13 22:03:20.031783 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:20.031761 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 13 22:03:20.047278 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:20.047243 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-hfxp7"] Jul 13 22:03:20.087410 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:20.087373 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpsvn\" (UniqueName: \"kubernetes.io/projected/a592e914-9dce-40aa-9af8-db424b124726-kube-api-access-zpsvn\") pod \"model-serving-api-54d89bd5fc-hfxp7\" (UID: \"a592e914-9dce-40aa-9af8-db424b124726\") " pod="opendatahub/model-serving-api-54d89bd5fc-hfxp7" Jul 13 22:03:20.087587 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:20.087422 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a592e914-9dce-40aa-9af8-db424b124726-tls-certs\") pod \"model-serving-api-54d89bd5fc-hfxp7\" (UID: \"a592e914-9dce-40aa-9af8-db424b124726\") " pod="opendatahub/model-serving-api-54d89bd5fc-hfxp7" Jul 13 22:03:20.187992 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:20.187965 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a592e914-9dce-40aa-9af8-db424b124726-tls-certs\") pod \"model-serving-api-54d89bd5fc-hfxp7\" (UID: \"a592e914-9dce-40aa-9af8-db424b124726\") " pod="opendatahub/model-serving-api-54d89bd5fc-hfxp7" Jul 13 22:03:20.188090 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:20.188023 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpsvn\" (UniqueName: \"kubernetes.io/projected/a592e914-9dce-40aa-9af8-db424b124726-kube-api-access-zpsvn\") pod \"model-serving-api-54d89bd5fc-hfxp7\" (UID: \"a592e914-9dce-40aa-9af8-db424b124726\") " pod="opendatahub/model-serving-api-54d89bd5fc-hfxp7" Jul 13 22:03:20.188144 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:03:20.188122 2569 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 13 22:03:20.188218 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:03:20.188208 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a592e914-9dce-40aa-9af8-db424b124726-tls-certs podName:a592e914-9dce-40aa-9af8-db424b124726 nodeName:}" failed. No retries permitted until 2026-07-13 22:03:20.688186572 +0000 UTC m=+313.345391319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a592e914-9dce-40aa-9af8-db424b124726-tls-certs") pod "model-serving-api-54d89bd5fc-hfxp7" (UID: "a592e914-9dce-40aa-9af8-db424b124726") : secret "model-serving-api-tls" not found Jul 13 22:03:20.200560 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:20.200520 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpsvn\" (UniqueName: \"kubernetes.io/projected/a592e914-9dce-40aa-9af8-db424b124726-kube-api-access-zpsvn\") pod \"model-serving-api-54d89bd5fc-hfxp7\" (UID: \"a592e914-9dce-40aa-9af8-db424b124726\") " pod="opendatahub/model-serving-api-54d89bd5fc-hfxp7" Jul 13 22:03:20.692262 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:20.692223 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a592e914-9dce-40aa-9af8-db424b124726-tls-certs\") pod \"model-serving-api-54d89bd5fc-hfxp7\" (UID: \"a592e914-9dce-40aa-9af8-db424b124726\") " pod="opendatahub/model-serving-api-54d89bd5fc-hfxp7" Jul 13 22:03:20.694598 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:20.694573 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a592e914-9dce-40aa-9af8-db424b124726-tls-certs\") pod \"model-serving-api-54d89bd5fc-hfxp7\" (UID: \"a592e914-9dce-40aa-9af8-db424b124726\") " pod="opendatahub/model-serving-api-54d89bd5fc-hfxp7" Jul 13 22:03:20.940105 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:20.940072 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-hfxp7" Jul 13 22:03:21.074644 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:21.074611 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-hfxp7"] Jul 13 22:03:21.076062 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:03:21.076032 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda592e914_9dce_40aa_9af8_db424b124726.slice/crio-9e9d8668ed700ee8f4be7060d7a4e4b11a6cf335e68816f48cfa1c9ef6cbb087 WatchSource:0}: Error finding container 9e9d8668ed700ee8f4be7060d7a4e4b11a6cf335e68816f48cfa1c9ef6cbb087: Status 404 returned error can't find the container with id 9e9d8668ed700ee8f4be7060d7a4e4b11a6cf335e68816f48cfa1c9ef6cbb087 Jul 13 22:03:21.077840 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:21.077824 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 22:03:21.502384 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:21.502340 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-hfxp7" event={"ID":"a592e914-9dce-40aa-9af8-db424b124726","Type":"ContainerStarted","Data":"9e9d8668ed700ee8f4be7060d7a4e4b11a6cf335e68816f48cfa1c9ef6cbb087"} Jul 13 22:03:23.511764 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:23.511729 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-hfxp7" event={"ID":"a592e914-9dce-40aa-9af8-db424b124726","Type":"ContainerStarted","Data":"93bd748aaef753a0cf8f631c45c0a423a976f13289dfd82f453bed670a44ff4a"} Jul 13 22:03:23.512200 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:23.511801 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-hfxp7" Jul 13 22:03:23.530499 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:23.530452 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-hfxp7" podStartSLOduration=1.501065497 podStartE2EDuration="3.530438571s" totalImagesPullingTime="2.029373074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:03:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:03:23.52840238 +0000 UTC m=+316.185607149" watchObservedRunningTime="2026-07-13 22:03:23.530438571 +0000 UTC m=+316.187643338" Jul 13 22:03:29.615569 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:29.615497 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-766bd547bb-8trqh"] Jul 13 22:03:29.618758 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:29.618740 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-766bd547bb-8trqh" Jul 13 22:03:29.633419 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:29.633401 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 13 22:03:29.633505 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:29.633425 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 13 22:03:29.657527 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:29.657503 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-766bd547bb-8trqh"] Jul 13 22:03:29.760961 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:29.760923 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d79pn\" (UniqueName: \"kubernetes.io/projected/0f150fe9-984a-482a-a204-59a4eb074dba-kube-api-access-d79pn\") pod \"kube-auth-proxy-766bd547bb-8trqh\" (UID: \"0f150fe9-984a-482a-a204-59a4eb074dba\") " pod="openshift-ingress/kube-auth-proxy-766bd547bb-8trqh" Jul 13 22:03:29.761087 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:29.760969 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f150fe9-984a-482a-a204-59a4eb074dba-tls-certs\") pod \"kube-auth-proxy-766bd547bb-8trqh\" (UID: \"0f150fe9-984a-482a-a204-59a4eb074dba\") " pod="openshift-ingress/kube-auth-proxy-766bd547bb-8trqh" Jul 13 22:03:29.761087 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:29.761015 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f150fe9-984a-482a-a204-59a4eb074dba-tmp\") pod \"kube-auth-proxy-766bd547bb-8trqh\" (UID: \"0f150fe9-984a-482a-a204-59a4eb074dba\") " pod="openshift-ingress/kube-auth-proxy-766bd547bb-8trqh" Jul 13 22:03:29.861814 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:29.861780 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d79pn\" (UniqueName: \"kubernetes.io/projected/0f150fe9-984a-482a-a204-59a4eb074dba-kube-api-access-d79pn\") pod \"kube-auth-proxy-766bd547bb-8trqh\" (UID: \"0f150fe9-984a-482a-a204-59a4eb074dba\") " pod="openshift-ingress/kube-auth-proxy-766bd547bb-8trqh" Jul 13 22:03:29.861940 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:29.861825 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f150fe9-984a-482a-a204-59a4eb074dba-tls-certs\") pod \"kube-auth-proxy-766bd547bb-8trqh\" (UID: \"0f150fe9-984a-482a-a204-59a4eb074dba\") " pod="openshift-ingress/kube-auth-proxy-766bd547bb-8trqh" Jul 13 22:03:29.861940 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:29.861852 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f150fe9-984a-482a-a204-59a4eb074dba-tmp\") pod \"kube-auth-proxy-766bd547bb-8trqh\" (UID: \"0f150fe9-984a-482a-a204-59a4eb074dba\") " pod="openshift-ingress/kube-auth-proxy-766bd547bb-8trqh" Jul 13 22:03:29.864160 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:29.864130 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f150fe9-984a-482a-a204-59a4eb074dba-tmp\") pod \"kube-auth-proxy-766bd547bb-8trqh\" (UID: \"0f150fe9-984a-482a-a204-59a4eb074dba\") " pod="openshift-ingress/kube-auth-proxy-766bd547bb-8trqh" Jul 13 22:03:29.864269 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:29.864227 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f150fe9-984a-482a-a204-59a4eb074dba-tls-certs\") pod \"kube-auth-proxy-766bd547bb-8trqh\" (UID: \"0f150fe9-984a-482a-a204-59a4eb074dba\") " pod="openshift-ingress/kube-auth-proxy-766bd547bb-8trqh" Jul 13 22:03:29.877236 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:29.877184 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d79pn\" (UniqueName: \"kubernetes.io/projected/0f150fe9-984a-482a-a204-59a4eb074dba-kube-api-access-d79pn\") pod \"kube-auth-proxy-766bd547bb-8trqh\" (UID: \"0f150fe9-984a-482a-a204-59a4eb074dba\") " pod="openshift-ingress/kube-auth-proxy-766bd547bb-8trqh" Jul 13 22:03:29.927880 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:29.927857 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-766bd547bb-8trqh" Jul 13 22:03:30.066518 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.066483 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d9k4k"] Jul 13 22:03:30.073569 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.072033 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d9k4k" Jul 13 22:03:30.076304 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.075888 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-2w9gv\"" Jul 13 22:03:30.076304 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.076121 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 13 22:03:30.084258 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.084232 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qdkhz"] Jul 13 22:03:30.088466 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.088433 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d9k4k"] Jul 13 22:03:30.088656 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.088558 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qdkhz" Jul 13 22:03:30.091002 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.090945 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-766bd547bb-8trqh"] Jul 13 22:03:30.091391 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.091152 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-tl7b5\"" Jul 13 22:03:30.091391 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.091208 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 13 22:03:30.095367 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:03:30.095339 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f150fe9_984a_482a_a204_59a4eb074dba.slice/crio-d842498761918e82f49868d6e7f00ed42a3e44b70a7a491a11b3dbd4b5010bcd WatchSource:0}: Error finding container d842498761918e82f49868d6e7f00ed42a3e44b70a7a491a11b3dbd4b5010bcd: Status 404 returned error can't find the container with id d842498761918e82f49868d6e7f00ed42a3e44b70a7a491a11b3dbd4b5010bcd Jul 13 22:03:30.111292 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.111215 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qdkhz"] Jul 13 22:03:30.163602 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.163508 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3955d6ec-f505-4205-beea-a28c95cc5b64-cert\") pod \"kserve-controller-manager-d6f4b65b8-d9k4k\" (UID: \"3955d6ec-f505-4205-beea-a28c95cc5b64\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d9k4k" Jul 13 22:03:30.163602 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.163556 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlgzw\" (UniqueName: \"kubernetes.io/projected/e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb-kube-api-access-rlgzw\") pod \"llmisvc-controller-manager-777c6bd7d5-qdkhz\" (UID: \"e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qdkhz" Jul 13 22:03:30.163798 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.163629 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr4s7\" (UniqueName: \"kubernetes.io/projected/3955d6ec-f505-4205-beea-a28c95cc5b64-kube-api-access-gr4s7\") pod \"kserve-controller-manager-d6f4b65b8-d9k4k\" (UID: \"3955d6ec-f505-4205-beea-a28c95cc5b64\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d9k4k" Jul 13 22:03:30.163798 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.163661 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qdkhz\" (UID: \"e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qdkhz" Jul 13 22:03:30.265081 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.265045 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3955d6ec-f505-4205-beea-a28c95cc5b64-cert\") pod \"kserve-controller-manager-d6f4b65b8-d9k4k\" (UID: \"3955d6ec-f505-4205-beea-a28c95cc5b64\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d9k4k" Jul 13 22:03:30.265272 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.265094 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlgzw\" (UniqueName: \"kubernetes.io/projected/e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb-kube-api-access-rlgzw\") pod \"llmisvc-controller-manager-777c6bd7d5-qdkhz\" (UID: \"e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qdkhz" Jul 13 22:03:30.265272 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.265139 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr4s7\" (UniqueName: \"kubernetes.io/projected/3955d6ec-f505-4205-beea-a28c95cc5b64-kube-api-access-gr4s7\") pod \"kserve-controller-manager-d6f4b65b8-d9k4k\" (UID: \"3955d6ec-f505-4205-beea-a28c95cc5b64\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d9k4k" Jul 13 22:03:30.265272 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.265171 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qdkhz\" (UID: \"e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qdkhz" Jul 13 22:03:30.265272 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:03:30.265208 2569 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 13 22:03:30.265272 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:03:30.265270 2569 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 13 22:03:30.265504 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:03:30.265294 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3955d6ec-f505-4205-beea-a28c95cc5b64-cert podName:3955d6ec-f505-4205-beea-a28c95cc5b64 nodeName:}" failed. No retries permitted until 2026-07-13 22:03:30.765265154 +0000 UTC m=+323.422469900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3955d6ec-f505-4205-beea-a28c95cc5b64-cert") pod "kserve-controller-manager-d6f4b65b8-d9k4k" (UID: "3955d6ec-f505-4205-beea-a28c95cc5b64") : secret "kserve-webhook-server-cert" not found Jul 13 22:03:30.265504 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:03:30.265324 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb-cert podName:e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb nodeName:}" failed. No retries permitted until 2026-07-13 22:03:30.765307713 +0000 UTC m=+323.422512465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb-cert") pod "llmisvc-controller-manager-777c6bd7d5-qdkhz" (UID: "e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb") : secret "llmisvc-webhook-server-cert" not found Jul 13 22:03:30.283633 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.283602 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlgzw\" (UniqueName: \"kubernetes.io/projected/e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb-kube-api-access-rlgzw\") pod \"llmisvc-controller-manager-777c6bd7d5-qdkhz\" (UID: \"e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qdkhz" Jul 13 22:03:30.283726 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.283680 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr4s7\" (UniqueName: \"kubernetes.io/projected/3955d6ec-f505-4205-beea-a28c95cc5b64-kube-api-access-gr4s7\") pod \"kserve-controller-manager-d6f4b65b8-d9k4k\" (UID: \"3955d6ec-f505-4205-beea-a28c95cc5b64\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d9k4k" Jul 13 22:03:30.541389 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.541353 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-766bd547bb-8trqh" event={"ID":"0f150fe9-984a-482a-a204-59a4eb074dba","Type":"ContainerStarted","Data":"d842498761918e82f49868d6e7f00ed42a3e44b70a7a491a11b3dbd4b5010bcd"} Jul 13 22:03:30.770173 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.770133 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3955d6ec-f505-4205-beea-a28c95cc5b64-cert\") pod \"kserve-controller-manager-d6f4b65b8-d9k4k\" (UID: \"3955d6ec-f505-4205-beea-a28c95cc5b64\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d9k4k" Jul 13 22:03:30.770567 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.770198 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qdkhz\" (UID: \"e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qdkhz" Jul 13 22:03:30.772590 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.772561 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qdkhz\" (UID: \"e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qdkhz" Jul 13 22:03:30.772693 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.772568 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3955d6ec-f505-4205-beea-a28c95cc5b64-cert\") pod \"kserve-controller-manager-d6f4b65b8-d9k4k\" (UID: \"3955d6ec-f505-4205-beea-a28c95cc5b64\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d9k4k" Jul 13 22:03:30.996693 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:30.996664 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d9k4k" Jul 13 22:03:31.004630 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:31.004600 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qdkhz" Jul 13 22:03:31.164577 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:31.164550 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d9k4k"] Jul 13 22:03:31.167444 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:03:31.167406 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3955d6ec_f505_4205_beea_a28c95cc5b64.slice/crio-b2cd7a072e5838090965fa5548b3bc0af5444ebc6cd88dd2a2b14c26c4a2bb46 WatchSource:0}: Error finding container b2cd7a072e5838090965fa5548b3bc0af5444ebc6cd88dd2a2b14c26c4a2bb46: Status 404 returned error can't find the container with id b2cd7a072e5838090965fa5548b3bc0af5444ebc6cd88dd2a2b14c26c4a2bb46 Jul 13 22:03:31.204728 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:31.204703 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qdkhz"] Jul 13 22:03:31.206443 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:03:31.206415 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode1ad7a64_ad39_40ca_99b2_f6f05c1cfecb.slice/crio-e9c8b65479a2622d5914fedb1a3118c78c26f48e6bf1455fe5debd24fb917ddb WatchSource:0}: Error finding container e9c8b65479a2622d5914fedb1a3118c78c26f48e6bf1455fe5debd24fb917ddb: Status 404 returned error can't find the container with id e9c8b65479a2622d5914fedb1a3118c78c26f48e6bf1455fe5debd24fb917ddb Jul 13 22:03:31.548994 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:31.548954 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qdkhz" event={"ID":"e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb","Type":"ContainerStarted","Data":"e9c8b65479a2622d5914fedb1a3118c78c26f48e6bf1455fe5debd24fb917ddb"} Jul 13 22:03:31.550045 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:31.550018 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d9k4k" event={"ID":"3955d6ec-f505-4205-beea-a28c95cc5b64","Type":"ContainerStarted","Data":"b2cd7a072e5838090965fa5548b3bc0af5444ebc6cd88dd2a2b14c26c4a2bb46"} Jul 13 22:03:34.521219 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:34.521185 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-hfxp7" Jul 13 22:03:35.571798 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:35.571761 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-766bd547bb-8trqh" event={"ID":"0f150fe9-984a-482a-a204-59a4eb074dba","Type":"ContainerStarted","Data":"9f8a98bde34f977325258bca76bb8fc3539957be72e8266fef428de588a0e9b6"} Jul 13 22:03:35.573269 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:35.573244 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qdkhz" event={"ID":"e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb","Type":"ContainerStarted","Data":"471cbe6f7dcd35ee0150562ee5aaf54d59c06694f1cc5c0e4b9acbe11f037ba2"} Jul 13 22:03:35.573401 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:35.573307 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qdkhz" Jul 13 22:03:35.574739 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:35.574720 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d9k4k" event={"ID":"3955d6ec-f505-4205-beea-a28c95cc5b64","Type":"ContainerStarted","Data":"805da7e36d9b86760d4f9b8bb3061dda209f877c56c877f0765661da79baca3a"} Jul 13 22:03:35.574823 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:35.574767 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d9k4k" Jul 13 22:03:35.590184 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:35.590145 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-766bd547bb-8trqh" podStartSLOduration=1.796276297 podStartE2EDuration="6.590132625s" totalImagesPullingTime="4.793856328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:03:35.589793352 +0000 UTC m=+328.246998121" watchObservedRunningTime="2026-07-13 22:03:35.590132625 +0000 UTC m=+328.247337392" Jul 13 22:03:35.608854 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:35.608808 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d9k4k" podStartSLOduration=1.886868079 podStartE2EDuration="5.608789071s" totalImagesPullingTime="3.721920992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:03:35.607173635 +0000 UTC m=+328.264378403" watchObservedRunningTime="2026-07-13 22:03:35.608789071 +0000 UTC m=+328.265993841" Jul 13 22:03:35.630510 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:35.630470 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qdkhz" podStartSLOduration=1.89509945 podStartE2EDuration="5.630456255s" totalImagesPullingTime="3.735356805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:03:35.629833345 +0000 UTC m=+328.287038114" watchObservedRunningTime="2026-07-13 22:03:35.630456255 +0000 UTC m=+328.287661024" Jul 13 22:03:38.437715 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.437685 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h"] Jul 13 22:03:38.441094 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.441076 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.444426 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.444407 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 13 22:03:38.444939 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.444917 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-wg9nm\"" Jul 13 22:03:38.464742 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.460377 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h"] Jul 13 22:03:38.537423 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.537392 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58111d7c-bdeb-40f7-b602-c23d9d496066-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.537423 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.537424 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58111d7c-bdeb-40f7-b602-c23d9d496066-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.537626 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.537444 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58111d7c-bdeb-40f7-b602-c23d9d496066-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.537626 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.537462 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmwkr\" (UniqueName: \"kubernetes.io/projected/58111d7c-bdeb-40f7-b602-c23d9d496066-kube-api-access-fmwkr\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.537626 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.537506 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58111d7c-bdeb-40f7-b602-c23d9d496066-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.537626 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.537567 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58111d7c-bdeb-40f7-b602-c23d9d496066-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.537626 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.537598 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58111d7c-bdeb-40f7-b602-c23d9d496066-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.537626 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.537616 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58111d7c-bdeb-40f7-b602-c23d9d496066-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.537833 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.537661 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58111d7c-bdeb-40f7-b602-c23d9d496066-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.638593 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.638566 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58111d7c-bdeb-40f7-b602-c23d9d496066-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.638708 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.638610 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58111d7c-bdeb-40f7-b602-c23d9d496066-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.638708 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.638634 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58111d7c-bdeb-40f7-b602-c23d9d496066-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.638708 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.638667 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58111d7c-bdeb-40f7-b602-c23d9d496066-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.638708 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.638702 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58111d7c-bdeb-40f7-b602-c23d9d496066-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.638909 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.638724 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58111d7c-bdeb-40f7-b602-c23d9d496066-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.638909 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.638751 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58111d7c-bdeb-40f7-b602-c23d9d496066-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.638909 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.638778 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmwkr\" (UniqueName: \"kubernetes.io/projected/58111d7c-bdeb-40f7-b602-c23d9d496066-kube-api-access-fmwkr\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.638909 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.638805 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58111d7c-bdeb-40f7-b602-c23d9d496066-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.639104 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.639013 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58111d7c-bdeb-40f7-b602-c23d9d496066-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.639219 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.639195 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58111d7c-bdeb-40f7-b602-c23d9d496066-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.639270 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.639230 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58111d7c-bdeb-40f7-b602-c23d9d496066-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.639451 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.639425 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58111d7c-bdeb-40f7-b602-c23d9d496066-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.639572 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.639429 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58111d7c-bdeb-40f7-b602-c23d9d496066-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.641052 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.641031 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58111d7c-bdeb-40f7-b602-c23d9d496066-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.641164 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.641067 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58111d7c-bdeb-40f7-b602-c23d9d496066-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.646893 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.646869 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58111d7c-bdeb-40f7-b602-c23d9d496066-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.647001 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.646986 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmwkr\" (UniqueName: \"kubernetes.io/projected/58111d7c-bdeb-40f7-b602-c23d9d496066-kube-api-access-fmwkr\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nr29h\" (UID: \"58111d7c-bdeb-40f7-b602-c23d9d496066\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.750700 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.750620 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:38.910833 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:38.910807 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h"] Jul 13 22:03:38.912742 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:03:38.912714 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58111d7c_bdeb_40f7_b602_c23d9d496066.slice/crio-021a30e65080247675d6910710d3f8751808013593e0e2be391cdd0dcf3f601d WatchSource:0}: Error finding container 021a30e65080247675d6910710d3f8751808013593e0e2be391cdd0dcf3f601d: Status 404 returned error can't find the container with id 021a30e65080247675d6910710d3f8751808013593e0e2be391cdd0dcf3f601d Jul 13 22:03:39.595326 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:39.595287 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" event={"ID":"58111d7c-bdeb-40f7-b602-c23d9d496066","Type":"ContainerStarted","Data":"021a30e65080247675d6910710d3f8751808013593e0e2be391cdd0dcf3f601d"} Jul 13 22:03:41.507185 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:41.507146 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:03:41.507471 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:41.507220 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:03:41.507471 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:41.507251 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:03:41.608077 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:41.608039 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" event={"ID":"58111d7c-bdeb-40f7-b602-c23d9d496066","Type":"ContainerStarted","Data":"ef60117a698698ebdc9b4bf0c24033822c5f571385b0d790f8a2b88ca5f82ac0"} Jul 13 22:03:41.629374 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:41.629325 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" podStartSLOduration=1.037113726 podStartE2EDuration="3.629309154s" totalImagesPullingTime="2.592195428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:03:41.627304871 +0000 UTC m=+334.284509642" watchObservedRunningTime="2026-07-13 22:03:41.629309154 +0000 UTC m=+334.286513921" Jul 13 22:03:41.751717 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:41.751690 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:42.755687 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:42.755660 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:43.617665 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:43.617638 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:03:43.618709 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:03:43.618679 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nr29h" Jul 13 22:04:06.581832 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:06.581752 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qdkhz" Jul 13 22:04:06.585117 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:06.585093 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d9k4k" Jul 13 22:04:20.020288 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.020256 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5blxg"] Jul 13 22:04:20.022646 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.022628 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5blxg" Jul 13 22:04:20.025344 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.025319 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 13 22:04:20.026292 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.026276 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-fvvxl\"" Jul 13 22:04:20.026292 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.026284 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 13 22:04:20.033517 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.033497 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5blxg"] Jul 13 22:04:20.049810 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.049780 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xhvj\" (UniqueName: \"kubernetes.io/projected/4df8a9fc-8b78-4204-a844-6519bf02c2ed-kube-api-access-5xhvj\") pod \"kuadrant-operator-catalog-5blxg\" (UID: \"4df8a9fc-8b78-4204-a844-6519bf02c2ed\") " pod="kuadrant-system/kuadrant-operator-catalog-5blxg" Jul 13 22:04:20.151001 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.150974 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xhvj\" (UniqueName: \"kubernetes.io/projected/4df8a9fc-8b78-4204-a844-6519bf02c2ed-kube-api-access-5xhvj\") pod \"kuadrant-operator-catalog-5blxg\" (UID: \"4df8a9fc-8b78-4204-a844-6519bf02c2ed\") " pod="kuadrant-system/kuadrant-operator-catalog-5blxg" Jul 13 22:04:20.158809 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.158789 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xhvj\" (UniqueName: \"kubernetes.io/projected/4df8a9fc-8b78-4204-a844-6519bf02c2ed-kube-api-access-5xhvj\") pod \"kuadrant-operator-catalog-5blxg\" (UID: \"4df8a9fc-8b78-4204-a844-6519bf02c2ed\") " pod="kuadrant-system/kuadrant-operator-catalog-5blxg" Jul 13 22:04:20.333668 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.333600 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5blxg" Jul 13 22:04:20.385577 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.385485 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5blxg"] Jul 13 22:04:20.485821 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.485795 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5blxg"] Jul 13 22:04:20.487624 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:04:20.487599 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df8a9fc_8b78_4204_a844_6519bf02c2ed.slice/crio-d76e8d0e80d12148d0494b30c384395da5ed126f4a23a3c044db1888bf2cdaa2 WatchSource:0}: Error finding container d76e8d0e80d12148d0494b30c384395da5ed126f4a23a3c044db1888bf2cdaa2: Status 404 returned error can't find the container with id d76e8d0e80d12148d0494b30c384395da5ed126f4a23a3c044db1888bf2cdaa2 Jul 13 22:04:20.595420 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.595352 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dj6np"] Jul 13 22:04:20.598161 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.598143 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dj6np" Jul 13 22:04:20.606697 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.606671 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dj6np"] Jul 13 22:04:20.655088 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.655052 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsxv7\" (UniqueName: \"kubernetes.io/projected/2ddaa225-99ef-4b36-b829-b8bae71be5b0-kube-api-access-hsxv7\") pod \"kuadrant-operator-catalog-dj6np\" (UID: \"2ddaa225-99ef-4b36-b829-b8bae71be5b0\") " pod="kuadrant-system/kuadrant-operator-catalog-dj6np" Jul 13 22:04:20.756336 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.756304 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsxv7\" (UniqueName: \"kubernetes.io/projected/2ddaa225-99ef-4b36-b829-b8bae71be5b0-kube-api-access-hsxv7\") pod \"kuadrant-operator-catalog-dj6np\" (UID: \"2ddaa225-99ef-4b36-b829-b8bae71be5b0\") " pod="kuadrant-system/kuadrant-operator-catalog-dj6np" Jul 13 22:04:20.764485 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.764458 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsxv7\" (UniqueName: \"kubernetes.io/projected/2ddaa225-99ef-4b36-b829-b8bae71be5b0-kube-api-access-hsxv7\") pod \"kuadrant-operator-catalog-dj6np\" (UID: \"2ddaa225-99ef-4b36-b829-b8bae71be5b0\") " pod="kuadrant-system/kuadrant-operator-catalog-dj6np" Jul 13 22:04:20.778255 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.778229 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5blxg" event={"ID":"4df8a9fc-8b78-4204-a844-6519bf02c2ed","Type":"ContainerStarted","Data":"d76e8d0e80d12148d0494b30c384395da5ed126f4a23a3c044db1888bf2cdaa2"} Jul 13 22:04:20.907835 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:20.907775 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dj6np" Jul 13 22:04:21.032979 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:21.032954 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dj6np"] Jul 13 22:04:21.035206 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:04:21.035181 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ddaa225_99ef_4b36_b829_b8bae71be5b0.slice/crio-64c34604917d868e8530c624889ba6df47a1288bb89c198e09f490b0234dbbea WatchSource:0}: Error finding container 64c34604917d868e8530c624889ba6df47a1288bb89c198e09f490b0234dbbea: Status 404 returned error can't find the container with id 64c34604917d868e8530c624889ba6df47a1288bb89c198e09f490b0234dbbea Jul 13 22:04:21.784875 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:21.784836 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dj6np" event={"ID":"2ddaa225-99ef-4b36-b829-b8bae71be5b0","Type":"ContainerStarted","Data":"64c34604917d868e8530c624889ba6df47a1288bb89c198e09f490b0234dbbea"} Jul 13 22:04:23.795210 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:23.795169 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5blxg" event={"ID":"4df8a9fc-8b78-4204-a844-6519bf02c2ed","Type":"ContainerStarted","Data":"16de592996d3dd7590f054e841234f218bcb6fdcf93d9e74b4aad2c4e6eb93ec"} Jul 13 22:04:23.795210 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:23.795185 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-5blxg" podUID="4df8a9fc-8b78-4204-a844-6519bf02c2ed" containerName="registry-server" containerID="cri-o://16de592996d3dd7590f054e841234f218bcb6fdcf93d9e74b4aad2c4e6eb93ec" gracePeriod=2 Jul 13 22:04:23.796819 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:23.796791 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dj6np" event={"ID":"2ddaa225-99ef-4b36-b829-b8bae71be5b0","Type":"ContainerStarted","Data":"ab8e4a76c1caaee88deb975c559ec1fb21a568f72aa3d73d6521e63de57837d3"} Jul 13 22:04:23.815802 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:23.815764 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-5blxg" podStartSLOduration=1.578090899 podStartE2EDuration="3.815750692s" totalImagesPullingTime="2.237659793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:04:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:04:23.814075638 +0000 UTC m=+376.471280406" watchObservedRunningTime="2026-07-13 22:04:23.815750692 +0000 UTC m=+376.472955459" Jul 13 22:04:23.833714 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:23.833673 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-dj6np" podStartSLOduration=2.140830946 podStartE2EDuration="3.833661038s" totalImagesPullingTime="1.692830092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:04:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:04:23.832472449 +0000 UTC m=+376.489677216" watchObservedRunningTime="2026-07-13 22:04:23.833661038 +0000 UTC m=+376.490865806" Jul 13 22:04:24.038911 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:24.038889 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5blxg" Jul 13 22:04:24.087431 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:24.087350 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xhvj\" (UniqueName: \"kubernetes.io/projected/4df8a9fc-8b78-4204-a844-6519bf02c2ed-kube-api-access-5xhvj\") pod \"4df8a9fc-8b78-4204-a844-6519bf02c2ed\" (UID: \"4df8a9fc-8b78-4204-a844-6519bf02c2ed\") " Jul 13 22:04:24.089444 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:24.089419 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df8a9fc-8b78-4204-a844-6519bf02c2ed-kube-api-access-5xhvj" (OuterVolumeSpecName: "kube-api-access-5xhvj") pod "4df8a9fc-8b78-4204-a844-6519bf02c2ed" (UID: "4df8a9fc-8b78-4204-a844-6519bf02c2ed"). InnerVolumeSpecName "kube-api-access-5xhvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:04:24.187842 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:24.187813 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xhvj\" (UniqueName: \"kubernetes.io/projected/4df8a9fc-8b78-4204-a844-6519bf02c2ed-kube-api-access-5xhvj\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:04:24.803342 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:24.803309 2569 generic.go:356] "Generic (PLEG): container finished" podID="4df8a9fc-8b78-4204-a844-6519bf02c2ed" containerID="16de592996d3dd7590f054e841234f218bcb6fdcf93d9e74b4aad2c4e6eb93ec" exitCode=0 Jul 13 22:04:24.803791 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:24.803354 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5blxg" Jul 13 22:04:24.803791 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:24.803398 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5blxg" event={"ID":"4df8a9fc-8b78-4204-a844-6519bf02c2ed","Type":"ContainerDied","Data":"16de592996d3dd7590f054e841234f218bcb6fdcf93d9e74b4aad2c4e6eb93ec"} Jul 13 22:04:24.803791 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:24.803446 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5blxg" event={"ID":"4df8a9fc-8b78-4204-a844-6519bf02c2ed","Type":"ContainerDied","Data":"d76e8d0e80d12148d0494b30c384395da5ed126f4a23a3c044db1888bf2cdaa2"} Jul 13 22:04:24.803791 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:24.803473 2569 scope.go:117] "RemoveContainer" containerID="16de592996d3dd7590f054e841234f218bcb6fdcf93d9e74b4aad2c4e6eb93ec" Jul 13 22:04:24.812734 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:24.812717 2569 scope.go:117] "RemoveContainer" containerID="16de592996d3dd7590f054e841234f218bcb6fdcf93d9e74b4aad2c4e6eb93ec" Jul 13 22:04:24.812997 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:04:24.812974 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16de592996d3dd7590f054e841234f218bcb6fdcf93d9e74b4aad2c4e6eb93ec\": container with ID starting with 16de592996d3dd7590f054e841234f218bcb6fdcf93d9e74b4aad2c4e6eb93ec not found: ID does not exist" containerID="16de592996d3dd7590f054e841234f218bcb6fdcf93d9e74b4aad2c4e6eb93ec" Jul 13 22:04:24.813057 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:24.813004 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16de592996d3dd7590f054e841234f218bcb6fdcf93d9e74b4aad2c4e6eb93ec"} err="failed to get container status \"16de592996d3dd7590f054e841234f218bcb6fdcf93d9e74b4aad2c4e6eb93ec\": rpc error: code = NotFound desc = could not find container \"16de592996d3dd7590f054e841234f218bcb6fdcf93d9e74b4aad2c4e6eb93ec\": container with ID starting with 16de592996d3dd7590f054e841234f218bcb6fdcf93d9e74b4aad2c4e6eb93ec not found: ID does not exist" Jul 13 22:04:24.824590 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:24.824527 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5blxg"] Jul 13 22:04:24.826105 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:24.826087 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5blxg"] Jul 13 22:04:25.930493 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:25.930462 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4df8a9fc-8b78-4204-a844-6519bf02c2ed" path="/var/lib/kubelet/pods/4df8a9fc-8b78-4204-a844-6519bf02c2ed/volumes" Jul 13 22:04:30.908214 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:30.908172 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-dj6np" Jul 13 22:04:30.908627 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:30.908231 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-dj6np" Jul 13 22:04:30.930641 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:30.930620 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-dj6np" Jul 13 22:04:31.857835 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:31.857804 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-dj6np" Jul 13 22:04:48.641918 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.641888 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-596f749758-2kl94"] Jul 13 22:04:48.642337 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.642224 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df8a9fc-8b78-4204-a844-6519bf02c2ed" containerName="registry-server" Jul 13 22:04:48.642337 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.642240 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df8a9fc-8b78-4204-a844-6519bf02c2ed" containerName="registry-server" Jul 13 22:04:48.642337 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.642301 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df8a9fc-8b78-4204-a844-6519bf02c2ed" containerName="registry-server" Jul 13 22:04:48.646246 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.646221 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.649604 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.649565 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 13 22:04:48.649751 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.649664 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d95cx\"" Jul 13 22:04:48.650708 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.650689 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 13 22:04:48.650803 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.650731 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 13 22:04:48.651740 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.651725 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 13 22:04:48.651842 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.651823 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 13 22:04:48.654919 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.654901 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 13 22:04:48.660618 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.660595 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-596f749758-2kl94"] Jul 13 22:04:48.772619 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.772590 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1f54888-1879-4e06-9465-d24bb404cce6-console-serving-cert\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.772750 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.772644 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1f54888-1879-4e06-9465-d24bb404cce6-console-oauth-config\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.772750 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.772715 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rxkb\" (UniqueName: \"kubernetes.io/projected/a1f54888-1879-4e06-9465-d24bb404cce6-kube-api-access-4rxkb\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.772838 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.772768 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1f54888-1879-4e06-9465-d24bb404cce6-oauth-serving-cert\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.772838 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.772790 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1f54888-1879-4e06-9465-d24bb404cce6-console-config\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.772838 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.772812 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1f54888-1879-4e06-9465-d24bb404cce6-trusted-ca-bundle\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.772838 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.772836 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f54888-1879-4e06-9465-d24bb404cce6-service-ca\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.873258 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.873230 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rxkb\" (UniqueName: \"kubernetes.io/projected/a1f54888-1879-4e06-9465-d24bb404cce6-kube-api-access-4rxkb\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.873396 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.873269 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1f54888-1879-4e06-9465-d24bb404cce6-oauth-serving-cert\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.873396 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.873288 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1f54888-1879-4e06-9465-d24bb404cce6-console-config\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.873396 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.873342 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1f54888-1879-4e06-9465-d24bb404cce6-trusted-ca-bundle\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.873396 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.873380 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f54888-1879-4e06-9465-d24bb404cce6-service-ca\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.873658 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.873427 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1f54888-1879-4e06-9465-d24bb404cce6-console-serving-cert\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.873658 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.873491 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1f54888-1879-4e06-9465-d24bb404cce6-console-oauth-config\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.874045 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.874017 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1f54888-1879-4e06-9465-d24bb404cce6-oauth-serving-cert\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.874153 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.874105 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1f54888-1879-4e06-9465-d24bb404cce6-console-config\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.874153 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.874125 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f54888-1879-4e06-9465-d24bb404cce6-service-ca\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.874421 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.874402 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1f54888-1879-4e06-9465-d24bb404cce6-trusted-ca-bundle\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.875838 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.875820 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1f54888-1879-4e06-9465-d24bb404cce6-console-oauth-config\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.875944 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.875929 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1f54888-1879-4e06-9465-d24bb404cce6-console-serving-cert\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.884342 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.884320 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rxkb\" (UniqueName: \"kubernetes.io/projected/a1f54888-1879-4e06-9465-d24bb404cce6-kube-api-access-4rxkb\") pod \"console-596f749758-2kl94\" (UID: \"a1f54888-1879-4e06-9465-d24bb404cce6\") " pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:48.955696 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:48.955676 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:49.112979 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:49.112946 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-596f749758-2kl94"] Jul 13 22:04:49.115362 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:04:49.115328 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f54888_1879_4e06_9465_d24bb404cce6.slice/crio-7173716e13b3b6bc8a2506eeb81e36b1885a97758082ac174291da1cd85e9a36 WatchSource:0}: Error finding container 7173716e13b3b6bc8a2506eeb81e36b1885a97758082ac174291da1cd85e9a36: Status 404 returned error can't find the container with id 7173716e13b3b6bc8a2506eeb81e36b1885a97758082ac174291da1cd85e9a36 Jul 13 22:04:49.923932 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:49.923892 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-596f749758-2kl94" event={"ID":"a1f54888-1879-4e06-9465-d24bb404cce6","Type":"ContainerStarted","Data":"2e2d0394f3196e827edeec23fdfac28a05040769556829b38b8c1423f0a4c41e"} Jul 13 22:04:49.924333 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:49.923950 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-596f749758-2kl94" event={"ID":"a1f54888-1879-4e06-9465-d24bb404cce6","Type":"ContainerStarted","Data":"7173716e13b3b6bc8a2506eeb81e36b1885a97758082ac174291da1cd85e9a36"} Jul 13 22:04:49.954827 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:49.954783 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-596f749758-2kl94" podStartSLOduration=1.954770133 podStartE2EDuration="1.954770133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:04:49.953238933 +0000 UTC m=+402.610443700" watchObservedRunningTime="2026-07-13 22:04:49.954770133 +0000 UTC m=+402.611974901" Jul 13 22:04:51.873955 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:51.871416 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-wwg6g"] Jul 13 22:04:51.876426 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:51.876283 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wwg6g" Jul 13 22:04:51.879935 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:51.879918 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-jwksg\"" Jul 13 22:04:51.880642 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:51.880619 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 13 22:04:51.903183 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:51.903159 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-wwg6g"] Jul 13 22:04:51.999388 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:51.999362 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz6m6\" (UniqueName: \"kubernetes.io/projected/ef6150b4-e3d0-4c32-bc04-6c491dd31172-kube-api-access-dz6m6\") pod \"dns-operator-controller-manager-864dd78c4c-wwg6g\" (UID: \"ef6150b4-e3d0-4c32-bc04-6c491dd31172\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wwg6g" Jul 13 22:04:52.100176 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:52.100152 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6m6\" (UniqueName: \"kubernetes.io/projected/ef6150b4-e3d0-4c32-bc04-6c491dd31172-kube-api-access-dz6m6\") pod \"dns-operator-controller-manager-864dd78c4c-wwg6g\" (UID: \"ef6150b4-e3d0-4c32-bc04-6c491dd31172\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wwg6g" Jul 13 22:04:52.119133 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:52.119103 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz6m6\" (UniqueName: \"kubernetes.io/projected/ef6150b4-e3d0-4c32-bc04-6c491dd31172-kube-api-access-dz6m6\") pod \"dns-operator-controller-manager-864dd78c4c-wwg6g\" (UID: \"ef6150b4-e3d0-4c32-bc04-6c491dd31172\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wwg6g" Jul 13 22:04:52.186572 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:52.186549 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wwg6g" Jul 13 22:04:52.333582 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:52.333558 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-wwg6g"] Jul 13 22:04:52.335845 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:04:52.335816 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6150b4_e3d0_4c32_bc04_6c491dd31172.slice/crio-3c08d6b4cd8262f84ca22396fa995d13beb47e8bbceb9da3946f26a6c6200e4c WatchSource:0}: Error finding container 3c08d6b4cd8262f84ca22396fa995d13beb47e8bbceb9da3946f26a6c6200e4c: Status 404 returned error can't find the container with id 3c08d6b4cd8262f84ca22396fa995d13beb47e8bbceb9da3946f26a6c6200e4c Jul 13 22:04:52.937449 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:52.937417 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wwg6g" event={"ID":"ef6150b4-e3d0-4c32-bc04-6c491dd31172","Type":"ContainerStarted","Data":"3c08d6b4cd8262f84ca22396fa995d13beb47e8bbceb9da3946f26a6c6200e4c"} Jul 13 22:04:54.949431 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:54.949398 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wwg6g" event={"ID":"ef6150b4-e3d0-4c32-bc04-6c491dd31172","Type":"ContainerStarted","Data":"d3e8b1e5a5a9322d7b511f9923be680f42bb132c17109616bade3ee55b30adc4"} Jul 13 22:04:54.949837 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:54.949448 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wwg6g" Jul 13 22:04:54.966400 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:54.966356 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wwg6g" podStartSLOduration=2.148282649 podStartE2EDuration="3.966342325s" totalImagesPullingTime="1.818059676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:04:54.965511168 +0000 UTC m=+407.622715935" watchObservedRunningTime="2026-07-13 22:04:54.966342325 +0000 UTC m=+407.623547093" Jul 13 22:04:55.967737 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:55.967705 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-l45vc"] Jul 13 22:04:55.971034 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:55.971017 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-l45vc" Jul 13 22:04:55.975135 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:55.975116 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-gk6cw\"" Jul 13 22:04:55.983863 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:55.983836 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-l45vc"] Jul 13 22:04:56.029761 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:56.029737 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmdnj\" (UniqueName: \"kubernetes.io/projected/712d72a6-274b-4f0f-b4a7-6c2c79820cac-kube-api-access-jmdnj\") pod \"authorino-operator-75cc8d5cb7-l45vc\" (UID: \"712d72a6-274b-4f0f-b4a7-6c2c79820cac\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-l45vc" Jul 13 22:04:56.130260 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:56.130231 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmdnj\" (UniqueName: \"kubernetes.io/projected/712d72a6-274b-4f0f-b4a7-6c2c79820cac-kube-api-access-jmdnj\") pod \"authorino-operator-75cc8d5cb7-l45vc\" (UID: \"712d72a6-274b-4f0f-b4a7-6c2c79820cac\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-l45vc" Jul 13 22:04:56.147202 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:56.147175 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmdnj\" (UniqueName: \"kubernetes.io/projected/712d72a6-274b-4f0f-b4a7-6c2c79820cac-kube-api-access-jmdnj\") pod \"authorino-operator-75cc8d5cb7-l45vc\" (UID: \"712d72a6-274b-4f0f-b4a7-6c2c79820cac\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-l45vc" Jul 13 22:04:56.281417 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:56.281361 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-l45vc" Jul 13 22:04:56.429391 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:56.429366 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-l45vc"] Jul 13 22:04:56.431148 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:04:56.431119 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712d72a6_274b_4f0f_b4a7_6c2c79820cac.slice/crio-a01d2ed1e8587fd8517280436a69a93e6731ed3b4ca4b2c1f9f8b23cc9d3c6c0 WatchSource:0}: Error finding container a01d2ed1e8587fd8517280436a69a93e6731ed3b4ca4b2c1f9f8b23cc9d3c6c0: Status 404 returned error can't find the container with id a01d2ed1e8587fd8517280436a69a93e6731ed3b4ca4b2c1f9f8b23cc9d3c6c0 Jul 13 22:04:56.960250 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:56.960215 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-l45vc" event={"ID":"712d72a6-274b-4f0f-b4a7-6c2c79820cac","Type":"ContainerStarted","Data":"a01d2ed1e8587fd8517280436a69a93e6731ed3b4ca4b2c1f9f8b23cc9d3c6c0"} Jul 13 22:04:58.956630 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:58.956602 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:58.956947 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:58.956647 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:58.961337 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:58.961318 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:58.970295 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:58.970264 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-l45vc" event={"ID":"712d72a6-274b-4f0f-b4a7-6c2c79820cac","Type":"ContainerStarted","Data":"eedab69c4badaf75e1741ea29d0c54ae450a236834696ab21e769d8d7bb6ef81"} Jul 13 22:04:58.970404 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:58.970384 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-l45vc" Jul 13 22:04:58.974294 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:58.974275 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-596f749758-2kl94" Jul 13 22:04:59.016550 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:04:59.016495 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-l45vc" podStartSLOduration=1.679108205 podStartE2EDuration="4.016482182s" totalImagesPullingTime="2.337373977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:04:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:04:59.01496724 +0000 UTC m=+411.672172010" watchObservedRunningTime="2026-07-13 22:04:59.016482182 +0000 UTC m=+411.673686949" Jul 13 22:05:03.862811 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:03.862777 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f"] Jul 13 22:05:03.866243 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:03.866225 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" Jul 13 22:05:03.868893 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:03.868874 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q69hx\"" Jul 13 22:05:03.891904 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:03.891885 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f"] Jul 13 22:05:04.006513 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:04.006483 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz6x2\" (UniqueName: \"kubernetes.io/projected/57874f81-5127-4584-be89-50a6bec7db51-kube-api-access-mz6x2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-spq9f\" (UID: \"57874f81-5127-4584-be89-50a6bec7db51\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" Jul 13 22:05:04.006653 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:04.006612 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/57874f81-5127-4584-be89-50a6bec7db51-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-spq9f\" (UID: \"57874f81-5127-4584-be89-50a6bec7db51\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" Jul 13 22:05:04.107978 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:04.107941 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/57874f81-5127-4584-be89-50a6bec7db51-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-spq9f\" (UID: \"57874f81-5127-4584-be89-50a6bec7db51\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" Jul 13 22:05:04.108144 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:04.108103 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz6x2\" (UniqueName: \"kubernetes.io/projected/57874f81-5127-4584-be89-50a6bec7db51-kube-api-access-mz6x2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-spq9f\" (UID: \"57874f81-5127-4584-be89-50a6bec7db51\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" Jul 13 22:05:04.108309 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:04.108290 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/57874f81-5127-4584-be89-50a6bec7db51-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-spq9f\" (UID: \"57874f81-5127-4584-be89-50a6bec7db51\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" Jul 13 22:05:04.116833 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:04.116782 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz6x2\" (UniqueName: \"kubernetes.io/projected/57874f81-5127-4584-be89-50a6bec7db51-kube-api-access-mz6x2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-spq9f\" (UID: \"57874f81-5127-4584-be89-50a6bec7db51\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" Jul 13 22:05:04.176426 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:04.176397 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" Jul 13 22:05:04.319934 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:04.319891 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f"] Jul 13 22:05:04.323522 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:05:04.323488 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57874f81_5127_4584_be89_50a6bec7db51.slice/crio-d2db885e8adccdb103f968404a2dac1b4253fd000eb626dd8243a8f5b6fd64fd WatchSource:0}: Error finding container d2db885e8adccdb103f968404a2dac1b4253fd000eb626dd8243a8f5b6fd64fd: Status 404 returned error can't find the container with id d2db885e8adccdb103f968404a2dac1b4253fd000eb626dd8243a8f5b6fd64fd Jul 13 22:05:04.999106 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:04.999069 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" event={"ID":"57874f81-5127-4584-be89-50a6bec7db51","Type":"ContainerStarted","Data":"d2db885e8adccdb103f968404a2dac1b4253fd000eb626dd8243a8f5b6fd64fd"} Jul 13 22:05:05.956413 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:05.956387 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wwg6g" Jul 13 22:05:09.977000 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:09.976971 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-l45vc" Jul 13 22:05:12.036130 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:12.036100 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" event={"ID":"57874f81-5127-4584-be89-50a6bec7db51","Type":"ContainerStarted","Data":"3071802342fb35af36c403a6b9e903f3c203ea5c6a45f3693ea4da08a34788ae"} Jul 13 22:05:12.036520 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:12.036222 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" Jul 13 22:05:12.056686 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:12.056642 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" podStartSLOduration=2.217728394 podStartE2EDuration="9.056629748s" totalImagesPullingTime="6.838901354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:05:12.054773018 +0000 UTC m=+424.711977785" watchObservedRunningTime="2026-07-13 22:05:12.056629748 +0000 UTC m=+424.713834516" Jul 13 22:05:23.043320 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:23.043290 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" Jul 13 22:05:24.705991 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.705957 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq"] Jul 13 22:05:24.711088 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.711071 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq" Jul 13 22:05:24.725170 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.725148 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq"] Jul 13 22:05:24.767036 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.767008 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f2hf\" (UniqueName: \"kubernetes.io/projected/06b412b0-7a61-4449-ac18-a64734d8d34a-kube-api-access-5f2hf\") pod \"kuadrant-operator-controller-manager-86878c769f-zkxwq\" (UID: \"06b412b0-7a61-4449-ac18-a64734d8d34a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq" Jul 13 22:05:24.767146 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.767070 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06b412b0-7a61-4449-ac18-a64734d8d34a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zkxwq\" (UID: \"06b412b0-7a61-4449-ac18-a64734d8d34a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq" Jul 13 22:05:24.867626 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.867603 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f2hf\" (UniqueName: \"kubernetes.io/projected/06b412b0-7a61-4449-ac18-a64734d8d34a-kube-api-access-5f2hf\") pod \"kuadrant-operator-controller-manager-86878c769f-zkxwq\" (UID: \"06b412b0-7a61-4449-ac18-a64734d8d34a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq" Jul 13 22:05:24.867849 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.867697 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06b412b0-7a61-4449-ac18-a64734d8d34a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zkxwq\" (UID: \"06b412b0-7a61-4449-ac18-a64734d8d34a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq" Jul 13 22:05:24.868077 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.868057 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06b412b0-7a61-4449-ac18-a64734d8d34a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zkxwq\" (UID: \"06b412b0-7a61-4449-ac18-a64734d8d34a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq" Jul 13 22:05:24.881787 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.881762 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f"] Jul 13 22:05:24.881930 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.881907 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" podUID="57874f81-5127-4584-be89-50a6bec7db51" containerName="manager" containerID="cri-o://3071802342fb35af36c403a6b9e903f3c203ea5c6a45f3693ea4da08a34788ae" gracePeriod=2 Jul 13 22:05:24.890314 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.890293 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f2hf\" (UniqueName: \"kubernetes.io/projected/06b412b0-7a61-4449-ac18-a64734d8d34a-kube-api-access-5f2hf\") pod \"kuadrant-operator-controller-manager-86878c769f-zkxwq\" (UID: \"06b412b0-7a61-4449-ac18-a64734d8d34a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq" Jul 13 22:05:24.893731 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.893705 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f"] Jul 13 22:05:24.901998 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:05:24.901868 2569 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"57874f81-5127-4584-be89-50a6bec7db51\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-spq9f\": pods \"kuadrant-operator-controller-manager-8684c9ff49-spq9f\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" Jul 13 22:05:24.912571 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.912550 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq"] Jul 13 22:05:24.913024 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.913005 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq" Jul 13 22:05:24.919318 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.919298 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq"] Jul 13 22:05:24.937396 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.937373 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5"] Jul 13 22:05:24.937790 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.937773 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57874f81-5127-4584-be89-50a6bec7db51" containerName="manager" Jul 13 22:05:24.937878 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.937793 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="57874f81-5127-4584-be89-50a6bec7db51" containerName="manager" Jul 13 22:05:24.937934 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.937917 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="57874f81-5127-4584-be89-50a6bec7db51" containerName="manager" Jul 13 22:05:24.941080 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.941062 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5" Jul 13 22:05:24.944130 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:05:24.944108 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-spq9f\" is forbidden: User \"system:node:ip-10-0-137-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-183.ec2.internal' and this object" podUID="57874f81-5127-4584-be89-50a6bec7db51" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" Jul 13 22:05:24.968689 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.968629 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/239b3fdd-1a24-44cb-9e56-78112f9f648b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sphb5\" (UID: \"239b3fdd-1a24-44cb-9e56-78112f9f648b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5" Jul 13 22:05:24.968777 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.968704 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p74xf\" (UniqueName: \"kubernetes.io/projected/239b3fdd-1a24-44cb-9e56-78112f9f648b-kube-api-access-p74xf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sphb5\" (UID: \"239b3fdd-1a24-44cb-9e56-78112f9f648b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5" Jul 13 22:05:24.978681 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.978658 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5"] Jul 13 22:05:24.980374 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.980350 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr"] Jul 13 22:05:24.983787 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:24.983770 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr" Jul 13 22:05:25.007824 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.007800 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr"] Jul 13 22:05:25.069797 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.069761 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p74xf\" (UniqueName: \"kubernetes.io/projected/239b3fdd-1a24-44cb-9e56-78112f9f648b-kube-api-access-p74xf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sphb5\" (UID: \"239b3fdd-1a24-44cb-9e56-78112f9f648b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5" Jul 13 22:05:25.069945 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.069845 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/239b3fdd-1a24-44cb-9e56-78112f9f648b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sphb5\" (UID: \"239b3fdd-1a24-44cb-9e56-78112f9f648b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5" Jul 13 22:05:25.070276 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.070251 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/239b3fdd-1a24-44cb-9e56-78112f9f648b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sphb5\" (UID: \"239b3fdd-1a24-44cb-9e56-78112f9f648b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5" Jul 13 22:05:25.085178 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.085147 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p74xf\" (UniqueName: \"kubernetes.io/projected/239b3fdd-1a24-44cb-9e56-78112f9f648b-kube-api-access-p74xf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sphb5\" (UID: \"239b3fdd-1a24-44cb-9e56-78112f9f648b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5" Jul 13 22:05:25.098858 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.098831 2569 generic.go:356] "Generic (PLEG): container finished" podID="57874f81-5127-4584-be89-50a6bec7db51" containerID="3071802342fb35af36c403a6b9e903f3c203ea5c6a45f3693ea4da08a34788ae" exitCode=0 Jul 13 22:05:25.118285 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.118264 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" Jul 13 22:05:25.137654 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:05:25.137625 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-spq9f\" is forbidden: User \"system:node:ip-10-0-137-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-183.ec2.internal' and this object" podUID="57874f81-5127-4584-be89-50a6bec7db51" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" Jul 13 22:05:25.142296 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:05:25.142274 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-spq9f\" is forbidden: User \"system:node:ip-10-0-137-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-183.ec2.internal' and this object" podUID="57874f81-5127-4584-be89-50a6bec7db51" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" Jul 13 22:05:25.170308 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.170286 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/57874f81-5127-4584-be89-50a6bec7db51-extensions-socket-volume\") pod \"57874f81-5127-4584-be89-50a6bec7db51\" (UID: \"57874f81-5127-4584-be89-50a6bec7db51\") " Jul 13 22:05:25.170402 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.170340 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz6x2\" (UniqueName: \"kubernetes.io/projected/57874f81-5127-4584-be89-50a6bec7db51-kube-api-access-mz6x2\") pod \"57874f81-5127-4584-be89-50a6bec7db51\" (UID: \"57874f81-5127-4584-be89-50a6bec7db51\") " Jul 13 22:05:25.170462 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.170446 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgbd5\" (UniqueName: \"kubernetes.io/projected/4ea9c003-5e7e-45b2-8f80-ca1b81068f89-kube-api-access-zgbd5\") pod \"kuadrant-operator-controller-manager-86878c769f-wzkxr\" (UID: \"4ea9c003-5e7e-45b2-8f80-ca1b81068f89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr" Jul 13 22:05:25.170519 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.170469 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ea9c003-5e7e-45b2-8f80-ca1b81068f89-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wzkxr\" (UID: \"4ea9c003-5e7e-45b2-8f80-ca1b81068f89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr" Jul 13 22:05:25.170775 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.170752 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57874f81-5127-4584-be89-50a6bec7db51-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "57874f81-5127-4584-be89-50a6bec7db51" (UID: "57874f81-5127-4584-be89-50a6bec7db51"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:05:25.172278 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.172258 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57874f81-5127-4584-be89-50a6bec7db51-kube-api-access-mz6x2" (OuterVolumeSpecName: "kube-api-access-mz6x2") pod "57874f81-5127-4584-be89-50a6bec7db51" (UID: "57874f81-5127-4584-be89-50a6bec7db51"). InnerVolumeSpecName "kube-api-access-mz6x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:05:25.271450 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.271388 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgbd5\" (UniqueName: \"kubernetes.io/projected/4ea9c003-5e7e-45b2-8f80-ca1b81068f89-kube-api-access-zgbd5\") pod \"kuadrant-operator-controller-manager-86878c769f-wzkxr\" (UID: \"4ea9c003-5e7e-45b2-8f80-ca1b81068f89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr" Jul 13 22:05:25.271450 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.271426 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ea9c003-5e7e-45b2-8f80-ca1b81068f89-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wzkxr\" (UID: \"4ea9c003-5e7e-45b2-8f80-ca1b81068f89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr" Jul 13 22:05:25.271636 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.271562 2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/57874f81-5127-4584-be89-50a6bec7db51-extensions-socket-volume\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:05:25.271636 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.271581 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz6x2\" (UniqueName: \"kubernetes.io/projected/57874f81-5127-4584-be89-50a6bec7db51-kube-api-access-mz6x2\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:05:25.271812 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.271792 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ea9c003-5e7e-45b2-8f80-ca1b81068f89-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wzkxr\" (UID: \"4ea9c003-5e7e-45b2-8f80-ca1b81068f89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr" Jul 13 22:05:25.281760 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.281743 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5" Jul 13 22:05:25.284852 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.284788 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgbd5\" (UniqueName: \"kubernetes.io/projected/4ea9c003-5e7e-45b2-8f80-ca1b81068f89-kube-api-access-zgbd5\") pod \"kuadrant-operator-controller-manager-86878c769f-wzkxr\" (UID: \"4ea9c003-5e7e-45b2-8f80-ca1b81068f89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr" Jul 13 22:05:25.293824 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.293807 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr" Jul 13 22:05:25.439025 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:05:25.438993 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239b3fdd_1a24_44cb_9e56_78112f9f648b.slice/crio-ba5294806a75ae31418c2935808d4bab9b419deb1ca03b1312157aa83832109c WatchSource:0}: Error finding container ba5294806a75ae31418c2935808d4bab9b419deb1ca03b1312157aa83832109c: Status 404 returned error can't find the container with id ba5294806a75ae31418c2935808d4bab9b419deb1ca03b1312157aa83832109c Jul 13 22:05:25.439214 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.439190 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5"] Jul 13 22:05:25.453134 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.453111 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr"] Jul 13 22:05:25.456819 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:05:25.456799 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea9c003_5e7e_45b2_8f80_ca1b81068f89.slice/crio-ed6b7140048d77ddb1da966a8f6c1e6233657c7994d3f2bcd79e6e71b6cc9a25 WatchSource:0}: Error finding container ed6b7140048d77ddb1da966a8f6c1e6233657c7994d3f2bcd79e6e71b6cc9a25: Status 404 returned error can't find the container with id ed6b7140048d77ddb1da966a8f6c1e6233657c7994d3f2bcd79e6e71b6cc9a25 Jul 13 22:05:25.931660 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:25.931628 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57874f81-5127-4584-be89-50a6bec7db51" path="/var/lib/kubelet/pods/57874f81-5127-4584-be89-50a6bec7db51/volumes" Jul 13 22:05:26.105845 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:26.105810 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr" event={"ID":"4ea9c003-5e7e-45b2-8f80-ca1b81068f89","Type":"ContainerStarted","Data":"689d4097bfc4bb007739f52093fc7abc367c6cf8219ad510fd9ed1cb7d5d6291"} Jul 13 22:05:26.106035 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:26.105963 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr" Jul 13 22:05:26.106035 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:26.106006 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr" event={"ID":"4ea9c003-5e7e-45b2-8f80-ca1b81068f89","Type":"ContainerStarted","Data":"ed6b7140048d77ddb1da966a8f6c1e6233657c7994d3f2bcd79e6e71b6cc9a25"} Jul 13 22:05:26.107108 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:26.107089 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-spq9f" Jul 13 22:05:26.107237 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:26.107140 2569 scope.go:117] "RemoveContainer" containerID="3071802342fb35af36c403a6b9e903f3c203ea5c6a45f3693ea4da08a34788ae" Jul 13 22:05:26.108833 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:26.108809 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5" event={"ID":"239b3fdd-1a24-44cb-9e56-78112f9f648b","Type":"ContainerStarted","Data":"65ff610c37f25df410782f0691804644b39f13e19514dc114b8b8ede3a1977b4"} Jul 13 22:05:26.108934 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:26.108862 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5" Jul 13 22:05:26.108934 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:26.108889 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5" event={"ID":"239b3fdd-1a24-44cb-9e56-78112f9f648b","Type":"ContainerStarted","Data":"ba5294806a75ae31418c2935808d4bab9b419deb1ca03b1312157aa83832109c"} Jul 13 22:05:26.138667 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:26.138628 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr" podStartSLOduration=2.138614808 podStartE2EDuration="2.138614808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:05:26.137903212 +0000 UTC m=+438.795107981" watchObservedRunningTime="2026-07-13 22:05:26.138614808 +0000 UTC m=+438.795819641" Jul 13 22:05:26.165956 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:26.165880 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5" podStartSLOduration=2.165866875 podStartE2EDuration="2.165866875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:05:26.159221158 +0000 UTC m=+438.816425929" watchObservedRunningTime="2026-07-13 22:05:26.165866875 +0000 UTC m=+438.823071642" Jul 13 22:05:27.460651 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:05:27.460608 2569 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-86878c769f-zkxwq_kuadrant-system_06b412b0-7a61-4449-ac18-a64734d8d34a_0(30020a89fe1ab665a415007c837172b8a84d27179f29b0e01ff9e878f7c4c82f): error adding pod kuadrant-system_kuadrant-operator-controller-manager-86878c769f-zkxwq to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 13 22:05:27.460957 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:05:27.460670 2569 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-86878c769f-zkxwq_kuadrant-system_06b412b0-7a61-4449-ac18-a64734d8d34a_0(30020a89fe1ab665a415007c837172b8a84d27179f29b0e01ff9e878f7c4c82f): error adding pod kuadrant-system_kuadrant-operator-controller-manager-86878c769f-zkxwq to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq" Jul 13 22:05:28.118444 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:28.118412 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq" Jul 13 22:05:28.120760 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:05:28.120730 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-zkxwq\" is forbidden: User \"system:node:ip-10-0-137-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-183.ec2.internal' and this object" podUID="06b412b0-7a61-4449-ac18-a64734d8d34a" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq" Jul 13 22:05:28.126040 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:28.126020 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq" Jul 13 22:05:28.128148 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:05:28.128118 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-zkxwq\" is forbidden: User \"system:node:ip-10-0-137-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-183.ec2.internal' and this object" podUID="06b412b0-7a61-4449-ac18-a64734d8d34a" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq" Jul 13 22:05:28.197785 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:28.197761 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f2hf\" (UniqueName: \"kubernetes.io/projected/06b412b0-7a61-4449-ac18-a64734d8d34a-kube-api-access-5f2hf\") pod \"06b412b0-7a61-4449-ac18-a64734d8d34a\" (UID: \"06b412b0-7a61-4449-ac18-a64734d8d34a\") " Jul 13 22:05:28.197879 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:28.197816 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06b412b0-7a61-4449-ac18-a64734d8d34a-extensions-socket-volume\") pod \"06b412b0-7a61-4449-ac18-a64734d8d34a\" (UID: \"06b412b0-7a61-4449-ac18-a64734d8d34a\") " Jul 13 22:05:28.198107 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:28.198082 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b412b0-7a61-4449-ac18-a64734d8d34a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "06b412b0-7a61-4449-ac18-a64734d8d34a" (UID: "06b412b0-7a61-4449-ac18-a64734d8d34a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:05:28.199736 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:28.199718 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b412b0-7a61-4449-ac18-a64734d8d34a-kube-api-access-5f2hf" (OuterVolumeSpecName: "kube-api-access-5f2hf") pod "06b412b0-7a61-4449-ac18-a64734d8d34a" (UID: "06b412b0-7a61-4449-ac18-a64734d8d34a"). InnerVolumeSpecName "kube-api-access-5f2hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:05:28.299465 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:28.299440 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f2hf\" (UniqueName: \"kubernetes.io/projected/06b412b0-7a61-4449-ac18-a64734d8d34a-kube-api-access-5f2hf\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:05:28.299465 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:28.299463 2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06b412b0-7a61-4449-ac18-a64734d8d34a-extensions-socket-volume\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:05:29.123185 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:29.123155 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq" Jul 13 22:05:29.125728 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:05:29.125693 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-zkxwq\" is forbidden: User \"system:node:ip-10-0-137-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-183.ec2.internal' and this object" podUID="06b412b0-7a61-4449-ac18-a64734d8d34a" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq" Jul 13 22:05:29.130931 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:05:29.130905 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-zkxwq\" is forbidden: User \"system:node:ip-10-0-137-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-183.ec2.internal' and this object" podUID="06b412b0-7a61-4449-ac18-a64734d8d34a" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zkxwq" Jul 13 22:05:29.930727 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:29.930694 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06b412b0-7a61-4449-ac18-a64734d8d34a" path="/var/lib/kubelet/pods/06b412b0-7a61-4449-ac18-a64734d8d34a/volumes" Jul 13 22:05:37.116345 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:37.116273 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr" Jul 13 22:05:37.116740 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:37.116656 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5" Jul 13 22:05:37.193872 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:37.193841 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5"] Jul 13 22:05:37.194035 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:37.194011 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5" podUID="239b3fdd-1a24-44cb-9e56-78112f9f648b" containerName="manager" containerID="cri-o://65ff610c37f25df410782f0691804644b39f13e19514dc114b8b8ede3a1977b4" gracePeriod=10 Jul 13 22:05:37.445971 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:37.445946 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5" Jul 13 22:05:37.476516 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:37.476432 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p74xf\" (UniqueName: \"kubernetes.io/projected/239b3fdd-1a24-44cb-9e56-78112f9f648b-kube-api-access-p74xf\") pod \"239b3fdd-1a24-44cb-9e56-78112f9f648b\" (UID: \"239b3fdd-1a24-44cb-9e56-78112f9f648b\") " Jul 13 22:05:37.476706 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:37.476561 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/239b3fdd-1a24-44cb-9e56-78112f9f648b-extensions-socket-volume\") pod \"239b3fdd-1a24-44cb-9e56-78112f9f648b\" (UID: \"239b3fdd-1a24-44cb-9e56-78112f9f648b\") " Jul 13 22:05:37.478006 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:37.477975 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239b3fdd-1a24-44cb-9e56-78112f9f648b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "239b3fdd-1a24-44cb-9e56-78112f9f648b" (UID: "239b3fdd-1a24-44cb-9e56-78112f9f648b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:05:37.480178 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:37.480151 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239b3fdd-1a24-44cb-9e56-78112f9f648b-kube-api-access-p74xf" (OuterVolumeSpecName: "kube-api-access-p74xf") pod "239b3fdd-1a24-44cb-9e56-78112f9f648b" (UID: "239b3fdd-1a24-44cb-9e56-78112f9f648b"). InnerVolumeSpecName "kube-api-access-p74xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:05:37.578652 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:37.578621 2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/239b3fdd-1a24-44cb-9e56-78112f9f648b-extensions-socket-volume\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:05:37.578652 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:37.578645 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p74xf\" (UniqueName: \"kubernetes.io/projected/239b3fdd-1a24-44cb-9e56-78112f9f648b-kube-api-access-p74xf\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:05:38.167833 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:38.167799 2569 generic.go:356] "Generic (PLEG): container finished" podID="239b3fdd-1a24-44cb-9e56-78112f9f648b" containerID="65ff610c37f25df410782f0691804644b39f13e19514dc114b8b8ede3a1977b4" exitCode=0 Jul 13 22:05:38.168281 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:38.167854 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5" Jul 13 22:05:38.168281 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:38.167870 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5" event={"ID":"239b3fdd-1a24-44cb-9e56-78112f9f648b","Type":"ContainerDied","Data":"65ff610c37f25df410782f0691804644b39f13e19514dc114b8b8ede3a1977b4"} Jul 13 22:05:38.168281 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:38.167911 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5" event={"ID":"239b3fdd-1a24-44cb-9e56-78112f9f648b","Type":"ContainerDied","Data":"ba5294806a75ae31418c2935808d4bab9b419deb1ca03b1312157aa83832109c"} Jul 13 22:05:38.168281 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:38.167932 2569 scope.go:117] "RemoveContainer" containerID="65ff610c37f25df410782f0691804644b39f13e19514dc114b8b8ede3a1977b4" Jul 13 22:05:38.176386 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:38.176367 2569 scope.go:117] "RemoveContainer" containerID="65ff610c37f25df410782f0691804644b39f13e19514dc114b8b8ede3a1977b4" Jul 13 22:05:38.176652 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:05:38.176636 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ff610c37f25df410782f0691804644b39f13e19514dc114b8b8ede3a1977b4\": container with ID starting with 65ff610c37f25df410782f0691804644b39f13e19514dc114b8b8ede3a1977b4 not found: ID does not exist" containerID="65ff610c37f25df410782f0691804644b39f13e19514dc114b8b8ede3a1977b4" Jul 13 22:05:38.176706 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:38.176661 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ff610c37f25df410782f0691804644b39f13e19514dc114b8b8ede3a1977b4"} err="failed to get container status \"65ff610c37f25df410782f0691804644b39f13e19514dc114b8b8ede3a1977b4\": rpc error: code = NotFound desc = could not find container \"65ff610c37f25df410782f0691804644b39f13e19514dc114b8b8ede3a1977b4\": container with ID starting with 65ff610c37f25df410782f0691804644b39f13e19514dc114b8b8ede3a1977b4 not found: ID does not exist" Jul 13 22:05:38.196523 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:38.196467 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5"] Jul 13 22:05:38.204395 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:38.204372 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sphb5"] Jul 13 22:05:39.931112 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:39.931083 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="239b3fdd-1a24-44cb-9e56-78112f9f648b" path="/var/lib/kubelet/pods/239b3fdd-1a24-44cb-9e56-78112f9f648b/volumes" Jul 13 22:05:53.561423 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.561388 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z"] Jul 13 22:05:53.561814 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.561794 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="239b3fdd-1a24-44cb-9e56-78112f9f648b" containerName="manager" Jul 13 22:05:53.561814 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.561807 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="239b3fdd-1a24-44cb-9e56-78112f9f648b" containerName="manager" Jul 13 22:05:53.561891 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.561882 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="239b3fdd-1a24-44cb-9e56-78112f9f648b" containerName="manager" Jul 13 22:05:53.564896 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.564876 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.567476 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.567456 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8g4n4\"" Jul 13 22:05:53.588613 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.588582 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z"] Jul 13 22:05:53.601243 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.601216 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.601414 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.601255 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx526\" (UniqueName: \"kubernetes.io/projected/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-kube-api-access-zx526\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.601414 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.601291 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.601414 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.601357 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.601414 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.601398 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.601659 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.601440 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.601659 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.601529 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.601659 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.601586 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.601659 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.601642 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.702806 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.702778 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.702962 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.702815 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.702962 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.702835 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.703069 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.702962 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.703069 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.703018 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.703069 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.703045 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx526\" (UniqueName: \"kubernetes.io/projected/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-kube-api-access-zx526\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.703228 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.703074 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.703228 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.703100 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.703228 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.703139 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.703228 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.703178 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.703426 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.703244 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.703484 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.703468 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.703582 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.703562 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.703679 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.703661 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.705422 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.705401 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.705582 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.705564 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.717889 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.717867 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.718030 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.718012 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx526\" (UniqueName: \"kubernetes.io/projected/f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3-kube-api-access-zx526\") pod \"maas-default-gateway-openshift-default-56654dcf7-cqx9z\" (UID: \"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:53.876372 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:53.876300 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:54.016589 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:54.016565 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z"] Jul 13 22:05:54.018129 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:05:54.018104 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91b6b4b_e207_4557_8a9a_1a4ee92ef4d3.slice/crio-bd240c66306e39bc1aac449393173bb02e55a21c3b825b45de46b0f0b965cb23 WatchSource:0}: Error finding container bd240c66306e39bc1aac449393173bb02e55a21c3b825b45de46b0f0b965cb23: Status 404 returned error can't find the container with id bd240c66306e39bc1aac449393173bb02e55a21c3b825b45de46b0f0b965cb23 Jul 13 22:05:54.020006 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:54.019979 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:05:54.020077 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:54.020039 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:05:54.020077 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:54.020067 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:05:54.245741 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:54.245709 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" event={"ID":"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3","Type":"ContainerStarted","Data":"30b1f1ee2470c994dd773fbf0064111e7e2edd75a434055272341e29dfdeb010"} Jul 13 22:05:54.246005 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:54.245759 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" event={"ID":"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3","Type":"ContainerStarted","Data":"bd240c66306e39bc1aac449393173bb02e55a21c3b825b45de46b0f0b965cb23"} Jul 13 22:05:54.267098 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:54.267044 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" podStartSLOduration=1.267027606 podStartE2EDuration="1.267027606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:05:54.265915661 +0000 UTC m=+466.923120432" watchObservedRunningTime="2026-07-13 22:05:54.267027606 +0000 UTC m=+466.924232373" Jul 13 22:05:54.877201 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:54.877171 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:55.882054 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:55.882019 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:56.094327 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:56.094297 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-kg5r9"] Jul 13 22:05:56.097984 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:56.097963 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kg5r9" Jul 13 22:05:56.100503 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:56.100478 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-89wkk\"" Jul 13 22:05:56.106352 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:56.106333 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-kg5r9"] Jul 13 22:05:56.222580 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:56.222527 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrsbg\" (UniqueName: \"kubernetes.io/projected/bce21c1e-05b5-492f-b069-21343129bc31-kube-api-access-mrsbg\") pod \"authorino-58cd955675-kg5r9\" (UID: \"bce21c1e-05b5-492f-b069-21343129bc31\") " pod="kuadrant-system/authorino-58cd955675-kg5r9" Jul 13 22:05:56.257137 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:56.257112 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:56.258167 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:56.258145 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:05:56.323553 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:56.323509 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrsbg\" (UniqueName: \"kubernetes.io/projected/bce21c1e-05b5-492f-b069-21343129bc31-kube-api-access-mrsbg\") pod \"authorino-58cd955675-kg5r9\" (UID: \"bce21c1e-05b5-492f-b069-21343129bc31\") " pod="kuadrant-system/authorino-58cd955675-kg5r9" Jul 13 22:05:56.331173 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:56.331148 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrsbg\" (UniqueName: \"kubernetes.io/projected/bce21c1e-05b5-492f-b069-21343129bc31-kube-api-access-mrsbg\") pod \"authorino-58cd955675-kg5r9\" (UID: \"bce21c1e-05b5-492f-b069-21343129bc31\") " pod="kuadrant-system/authorino-58cd955675-kg5r9" Jul 13 22:05:56.407911 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:56.407885 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kg5r9" Jul 13 22:05:56.536952 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:56.536928 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-kg5r9"] Jul 13 22:05:56.539226 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:05:56.539197 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbce21c1e_05b5_492f_b069_21343129bc31.slice/crio-be0559c9f941e1e73dc26855a4e0dcf85d199a8ffbf77f05382c90b3db21e4b9 WatchSource:0}: Error finding container be0559c9f941e1e73dc26855a4e0dcf85d199a8ffbf77f05382c90b3db21e4b9: Status 404 returned error can't find the container with id be0559c9f941e1e73dc26855a4e0dcf85d199a8ffbf77f05382c90b3db21e4b9 Jul 13 22:05:57.264771 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:05:57.264707 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kg5r9" event={"ID":"bce21c1e-05b5-492f-b069-21343129bc31","Type":"ContainerStarted","Data":"be0559c9f941e1e73dc26855a4e0dcf85d199a8ffbf77f05382c90b3db21e4b9"} Jul 13 22:06:00.284640 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:00.284589 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kg5r9" event={"ID":"bce21c1e-05b5-492f-b069-21343129bc31","Type":"ContainerStarted","Data":"b7d1675f58ac2411831c146adfd5bfc4948bda48b84cb8f017178f3e495e18e6"} Jul 13 22:06:00.306412 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:00.306364 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-kg5r9" podStartSLOduration=1.011979453 podStartE2EDuration="4.306349517s" totalImagesPullingTime="3.294370064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:05:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:06:00.304477724 +0000 UTC m=+472.961682483" watchObservedRunningTime="2026-07-13 22:06:00.306349517 +0000 UTC m=+472.963554284" Jul 13 22:06:02.622850 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:02.622817 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-kg5r9"] Jul 13 22:06:02.623317 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:02.622973 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-kg5r9" podUID="bce21c1e-05b5-492f-b069-21343129bc31" containerName="authorino" containerID="cri-o://b7d1675f58ac2411831c146adfd5bfc4948bda48b84cb8f017178f3e495e18e6" gracePeriod=30 Jul 13 22:06:02.870493 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:02.870466 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kg5r9" Jul 13 22:06:02.986327 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:02.986300 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrsbg\" (UniqueName: \"kubernetes.io/projected/bce21c1e-05b5-492f-b069-21343129bc31-kube-api-access-mrsbg\") pod \"bce21c1e-05b5-492f-b069-21343129bc31\" (UID: \"bce21c1e-05b5-492f-b069-21343129bc31\") " Jul 13 22:06:02.988214 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:02.988186 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce21c1e-05b5-492f-b069-21343129bc31-kube-api-access-mrsbg" (OuterVolumeSpecName: "kube-api-access-mrsbg") pod "bce21c1e-05b5-492f-b069-21343129bc31" (UID: "bce21c1e-05b5-492f-b069-21343129bc31"). InnerVolumeSpecName "kube-api-access-mrsbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:06:03.086926 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:03.086892 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrsbg\" (UniqueName: \"kubernetes.io/projected/bce21c1e-05b5-492f-b069-21343129bc31-kube-api-access-mrsbg\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:06:03.305400 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:03.305316 2569 generic.go:356] "Generic (PLEG): container finished" podID="bce21c1e-05b5-492f-b069-21343129bc31" containerID="b7d1675f58ac2411831c146adfd5bfc4948bda48b84cb8f017178f3e495e18e6" exitCode=0 Jul 13 22:06:03.305400 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:03.305371 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kg5r9" Jul 13 22:06:03.305601 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:03.305395 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kg5r9" event={"ID":"bce21c1e-05b5-492f-b069-21343129bc31","Type":"ContainerDied","Data":"b7d1675f58ac2411831c146adfd5bfc4948bda48b84cb8f017178f3e495e18e6"} Jul 13 22:06:03.305601 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:03.305435 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kg5r9" event={"ID":"bce21c1e-05b5-492f-b069-21343129bc31","Type":"ContainerDied","Data":"be0559c9f941e1e73dc26855a4e0dcf85d199a8ffbf77f05382c90b3db21e4b9"} Jul 13 22:06:03.305601 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:03.305452 2569 scope.go:117] "RemoveContainer" containerID="b7d1675f58ac2411831c146adfd5bfc4948bda48b84cb8f017178f3e495e18e6" Jul 13 22:06:03.314284 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:03.314267 2569 scope.go:117] "RemoveContainer" containerID="b7d1675f58ac2411831c146adfd5bfc4948bda48b84cb8f017178f3e495e18e6" Jul 13 22:06:03.314526 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:06:03.314508 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d1675f58ac2411831c146adfd5bfc4948bda48b84cb8f017178f3e495e18e6\": container with ID starting with b7d1675f58ac2411831c146adfd5bfc4948bda48b84cb8f017178f3e495e18e6 not found: ID does not exist" containerID="b7d1675f58ac2411831c146adfd5bfc4948bda48b84cb8f017178f3e495e18e6" Jul 13 22:06:03.314643 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:03.314551 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d1675f58ac2411831c146adfd5bfc4948bda48b84cb8f017178f3e495e18e6"} err="failed to get container status \"b7d1675f58ac2411831c146adfd5bfc4948bda48b84cb8f017178f3e495e18e6\": rpc error: code = NotFound desc = could not find container \"b7d1675f58ac2411831c146adfd5bfc4948bda48b84cb8f017178f3e495e18e6\": container with ID starting with b7d1675f58ac2411831c146adfd5bfc4948bda48b84cb8f017178f3e495e18e6 not found: ID does not exist" Jul 13 22:06:03.327785 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:03.327763 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-kg5r9"] Jul 13 22:06:03.334447 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:03.334428 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-kg5r9"] Jul 13 22:06:03.930509 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:03.930478 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bce21c1e-05b5-492f-b069-21343129bc31" path="/var/lib/kubelet/pods/bce21c1e-05b5-492f-b069-21343129bc31/volumes" Jul 13 22:06:29.612425 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:29.612391 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-59c4r"] Jul 13 22:06:29.612904 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:29.612775 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce21c1e-05b5-492f-b069-21343129bc31" containerName="authorino" Jul 13 22:06:29.612904 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:29.612787 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce21c1e-05b5-492f-b069-21343129bc31" containerName="authorino" Jul 13 22:06:29.612904 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:29.612864 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bce21c1e-05b5-492f-b069-21343129bc31" containerName="authorino" Jul 13 22:06:29.617511 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:29.617491 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-59c4r" Jul 13 22:06:29.620212 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:29.620191 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-89wkk\"" Jul 13 22:06:29.624918 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:29.624899 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-59c4r"] Jul 13 22:06:29.796606 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:29.796578 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7r5c\" (UniqueName: \"kubernetes.io/projected/4daa3cc3-c468-43e1-a556-04f33511a1a5-kube-api-access-g7r5c\") pod \"authorino-5d4b67c446-59c4r\" (UID: \"4daa3cc3-c468-43e1-a556-04f33511a1a5\") " pod="kuadrant-system/authorino-5d4b67c446-59c4r" Jul 13 22:06:29.851125 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:29.851095 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-59c4r"] Jul 13 22:06:29.851319 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:06:29.851300 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-g7r5c], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-59c4r" podUID="4daa3cc3-c468-43e1-a556-04f33511a1a5" Jul 13 22:06:29.897584 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:29.897501 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7r5c\" (UniqueName: \"kubernetes.io/projected/4daa3cc3-c468-43e1-a556-04f33511a1a5-kube-api-access-g7r5c\") pod \"authorino-5d4b67c446-59c4r\" (UID: \"4daa3cc3-c468-43e1-a556-04f33511a1a5\") " pod="kuadrant-system/authorino-5d4b67c446-59c4r" Jul 13 22:06:29.900057 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:29.900038 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d47576bc5-fvrcx"] Jul 13 22:06:29.903423 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:29.903408 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d47576bc5-fvrcx" Jul 13 22:06:29.906182 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:29.906165 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 13 22:06:29.913963 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:29.913944 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d47576bc5-fvrcx"] Jul 13 22:06:29.922808 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:29.922784 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7r5c\" (UniqueName: \"kubernetes.io/projected/4daa3cc3-c468-43e1-a556-04f33511a1a5-kube-api-access-g7r5c\") pod \"authorino-5d4b67c446-59c4r\" (UID: \"4daa3cc3-c468-43e1-a556-04f33511a1a5\") " pod="kuadrant-system/authorino-5d4b67c446-59c4r" Jul 13 22:06:29.998393 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:29.998352 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzw5v\" (UniqueName: \"kubernetes.io/projected/e0a4ec79-e40c-4ed5-88c0-d46d721add18-kube-api-access-fzw5v\") pod \"authorino-6d47576bc5-fvrcx\" (UID: \"e0a4ec79-e40c-4ed5-88c0-d46d721add18\") " pod="kuadrant-system/authorino-6d47576bc5-fvrcx" Jul 13 22:06:29.998562 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:29.998421 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e0a4ec79-e40c-4ed5-88c0-d46d721add18-tls-cert\") pod \"authorino-6d47576bc5-fvrcx\" (UID: \"e0a4ec79-e40c-4ed5-88c0-d46d721add18\") " pod="kuadrant-system/authorino-6d47576bc5-fvrcx" Jul 13 22:06:30.099864 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:30.099831 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzw5v\" (UniqueName: \"kubernetes.io/projected/e0a4ec79-e40c-4ed5-88c0-d46d721add18-kube-api-access-fzw5v\") pod \"authorino-6d47576bc5-fvrcx\" (UID: \"e0a4ec79-e40c-4ed5-88c0-d46d721add18\") " pod="kuadrant-system/authorino-6d47576bc5-fvrcx" Jul 13 22:06:30.099991 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:30.099889 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e0a4ec79-e40c-4ed5-88c0-d46d721add18-tls-cert\") pod \"authorino-6d47576bc5-fvrcx\" (UID: \"e0a4ec79-e40c-4ed5-88c0-d46d721add18\") " pod="kuadrant-system/authorino-6d47576bc5-fvrcx" Jul 13 22:06:30.102344 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:30.102318 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e0a4ec79-e40c-4ed5-88c0-d46d721add18-tls-cert\") pod \"authorino-6d47576bc5-fvrcx\" (UID: \"e0a4ec79-e40c-4ed5-88c0-d46d721add18\") " pod="kuadrant-system/authorino-6d47576bc5-fvrcx" Jul 13 22:06:30.112002 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:30.111979 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzw5v\" (UniqueName: \"kubernetes.io/projected/e0a4ec79-e40c-4ed5-88c0-d46d721add18-kube-api-access-fzw5v\") pod \"authorino-6d47576bc5-fvrcx\" (UID: \"e0a4ec79-e40c-4ed5-88c0-d46d721add18\") " pod="kuadrant-system/authorino-6d47576bc5-fvrcx" Jul 13 22:06:30.212871 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:30.212842 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d47576bc5-fvrcx" Jul 13 22:06:30.349042 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:30.348921 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d47576bc5-fvrcx"] Jul 13 22:06:30.351280 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:06:30.351252 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0a4ec79_e40c_4ed5_88c0_d46d721add18.slice/crio-8b3c2a07a591d8532edc764b6fec06c1b0ff393a63a8576a13f7fa4761068dd6 WatchSource:0}: Error finding container 8b3c2a07a591d8532edc764b6fec06c1b0ff393a63a8576a13f7fa4761068dd6: Status 404 returned error can't find the container with id 8b3c2a07a591d8532edc764b6fec06c1b0ff393a63a8576a13f7fa4761068dd6 Jul 13 22:06:30.440727 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:30.440695 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d47576bc5-fvrcx" event={"ID":"e0a4ec79-e40c-4ed5-88c0-d46d721add18","Type":"ContainerStarted","Data":"8b3c2a07a591d8532edc764b6fec06c1b0ff393a63a8576a13f7fa4761068dd6"} Jul 13 22:06:30.440827 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:30.440743 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-59c4r" Jul 13 22:06:30.448674 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:30.448657 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-59c4r" Jul 13 22:06:30.604882 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:30.604818 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7r5c\" (UniqueName: \"kubernetes.io/projected/4daa3cc3-c468-43e1-a556-04f33511a1a5-kube-api-access-g7r5c\") pod \"4daa3cc3-c468-43e1-a556-04f33511a1a5\" (UID: \"4daa3cc3-c468-43e1-a556-04f33511a1a5\") " Jul 13 22:06:30.606839 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:30.606811 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4daa3cc3-c468-43e1-a556-04f33511a1a5-kube-api-access-g7r5c" (OuterVolumeSpecName: "kube-api-access-g7r5c") pod "4daa3cc3-c468-43e1-a556-04f33511a1a5" (UID: "4daa3cc3-c468-43e1-a556-04f33511a1a5"). InnerVolumeSpecName "kube-api-access-g7r5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:06:30.705513 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:30.705493 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7r5c\" (UniqueName: \"kubernetes.io/projected/4daa3cc3-c468-43e1-a556-04f33511a1a5-kube-api-access-g7r5c\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:06:31.447369 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:31.447338 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-59c4r" Jul 13 22:06:31.447577 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:31.447331 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d47576bc5-fvrcx" event={"ID":"e0a4ec79-e40c-4ed5-88c0-d46d721add18","Type":"ContainerStarted","Data":"455ffc282a3407347e58b0bcf01a4aef6f6c975ec98c3570bac87549e1b8abd9"} Jul 13 22:06:31.468031 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:31.467989 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d47576bc5-fvrcx" podStartSLOduration=2.117356899 podStartE2EDuration="2.467974542s" totalImagesPullingTime="350.617643ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:06:31.467680555 +0000 UTC m=+504.124885322" watchObservedRunningTime="2026-07-13 22:06:31.467974542 +0000 UTC m=+504.125179367" Jul 13 22:06:31.492791 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:31.492760 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-59c4r"] Jul 13 22:06:31.504336 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:31.504306 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-59c4r"] Jul 13 22:06:31.931731 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:06:31.931658 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4daa3cc3-c468-43e1-a556-04f33511a1a5" path="/var/lib/kubelet/pods/4daa3cc3-c468-43e1-a556-04f33511a1a5/volumes" Jul 13 22:07:14.162660 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:14.162585 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n"] Jul 13 22:07:14.165960 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:14.165942 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n" Jul 13 22:07:14.169124 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:14.169106 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 13 22:07:14.169642 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:14.169627 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 22:07:14.169744 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:14.169723 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 13 22:07:14.170482 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:14.170467 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lzf6t\"" Jul 13 22:07:14.183359 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:14.183335 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n"] Jul 13 22:07:14.239061 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:14.239032 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f653da18-9dcf-4511-b761-5f6a5b705c2b-openshift-service-ca\") pod \"maas-api-75cffb6f44-qnk2n\" (UID: \"f653da18-9dcf-4511-b761-5f6a5b705c2b\") " pod="odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n" Jul 13 22:07:14.239167 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:14.239074 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqn48\" (UniqueName: \"kubernetes.io/projected/f653da18-9dcf-4511-b761-5f6a5b705c2b-kube-api-access-xqn48\") pod \"maas-api-75cffb6f44-qnk2n\" (UID: \"f653da18-9dcf-4511-b761-5f6a5b705c2b\") " pod="odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n" Jul 13 22:07:14.239167 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:14.239095 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f653da18-9dcf-4511-b761-5f6a5b705c2b-maas-api-tls\") pod \"maas-api-75cffb6f44-qnk2n\" (UID: \"f653da18-9dcf-4511-b761-5f6a5b705c2b\") " pod="odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n" Jul 13 22:07:14.340045 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:14.340019 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f653da18-9dcf-4511-b761-5f6a5b705c2b-openshift-service-ca\") pod \"maas-api-75cffb6f44-qnk2n\" (UID: \"f653da18-9dcf-4511-b761-5f6a5b705c2b\") " pod="odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n" Jul 13 22:07:14.340183 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:14.340060 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqn48\" (UniqueName: \"kubernetes.io/projected/f653da18-9dcf-4511-b761-5f6a5b705c2b-kube-api-access-xqn48\") pod \"maas-api-75cffb6f44-qnk2n\" (UID: \"f653da18-9dcf-4511-b761-5f6a5b705c2b\") " pod="odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n" Jul 13 22:07:14.340183 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:14.340091 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f653da18-9dcf-4511-b761-5f6a5b705c2b-maas-api-tls\") pod \"maas-api-75cffb6f44-qnk2n\" (UID: \"f653da18-9dcf-4511-b761-5f6a5b705c2b\") " pod="odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n" Jul 13 22:07:14.340293 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:07:14.340207 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 13 22:07:14.340293 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:07:14.340281 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f653da18-9dcf-4511-b761-5f6a5b705c2b-maas-api-tls podName:f653da18-9dcf-4511-b761-5f6a5b705c2b nodeName:}" failed. No retries permitted until 2026-07-13 22:07:14.840260894 +0000 UTC m=+547.497465643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f653da18-9dcf-4511-b761-5f6a5b705c2b-maas-api-tls") pod "maas-api-75cffb6f44-qnk2n" (UID: "f653da18-9dcf-4511-b761-5f6a5b705c2b") : secret "maas-api-serving-cert" not found Jul 13 22:07:14.340740 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:14.340721 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f653da18-9dcf-4511-b761-5f6a5b705c2b-openshift-service-ca\") pod \"maas-api-75cffb6f44-qnk2n\" (UID: \"f653da18-9dcf-4511-b761-5f6a5b705c2b\") " pod="odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n" Jul 13 22:07:14.353453 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:14.353430 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqn48\" (UniqueName: \"kubernetes.io/projected/f653da18-9dcf-4511-b761-5f6a5b705c2b-kube-api-access-xqn48\") pod \"maas-api-75cffb6f44-qnk2n\" (UID: \"f653da18-9dcf-4511-b761-5f6a5b705c2b\") " pod="odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n" Jul 13 22:07:14.844660 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:14.844629 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f653da18-9dcf-4511-b761-5f6a5b705c2b-maas-api-tls\") pod \"maas-api-75cffb6f44-qnk2n\" (UID: \"f653da18-9dcf-4511-b761-5f6a5b705c2b\") " pod="odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n" Jul 13 22:07:14.846995 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:14.846975 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f653da18-9dcf-4511-b761-5f6a5b705c2b-maas-api-tls\") pod \"maas-api-75cffb6f44-qnk2n\" (UID: \"f653da18-9dcf-4511-b761-5f6a5b705c2b\") " pod="odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n" Jul 13 22:07:15.076718 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:15.076684 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n" Jul 13 22:07:15.230136 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:15.229993 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n"] Jul 13 22:07:15.232697 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:07:15.232669 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf653da18_9dcf_4511_b761_5f6a5b705c2b.slice/crio-c618b1d546d5f202115cbb454668853bba083baa16b64802ffa07453fa119e00 WatchSource:0}: Error finding container c618b1d546d5f202115cbb454668853bba083baa16b64802ffa07453fa119e00: Status 404 returned error can't find the container with id c618b1d546d5f202115cbb454668853bba083baa16b64802ffa07453fa119e00 Jul 13 22:07:15.659944 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:15.659908 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n" event={"ID":"f653da18-9dcf-4511-b761-5f6a5b705c2b","Type":"ContainerStarted","Data":"c618b1d546d5f202115cbb454668853bba083baa16b64802ffa07453fa119e00"} Jul 13 22:07:18.675892 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:18.675859 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n" event={"ID":"f653da18-9dcf-4511-b761-5f6a5b705c2b","Type":"ContainerStarted","Data":"0c6bb0391920448e800b1a24e32014ac23a9a7df466883d11f3a9b118de2c17f"} Jul 13 22:07:18.676266 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:18.675943 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n" Jul 13 22:07:18.698003 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:18.697950 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n" podStartSLOduration=1.811585239 podStartE2EDuration="4.69791592s" totalImagesPullingTime="2.886330681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:07:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:07:18.695278617 +0000 UTC m=+551.352483386" watchObservedRunningTime="2026-07-13 22:07:18.69791592 +0000 UTC m=+551.355120690" Jul 13 22:07:22.700485 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:22.700451 2569 generic.go:356] "Generic (PLEG): container finished" podID="f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3" containerID="30b1f1ee2470c994dd773fbf0064111e7e2edd75a434055272341e29dfdeb010" exitCode=0 Jul 13 22:07:22.700897 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:22.700527 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" event={"ID":"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3","Type":"ContainerDied","Data":"30b1f1ee2470c994dd773fbf0064111e7e2edd75a434055272341e29dfdeb010"} Jul 13 22:07:22.700939 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:22.700896 2569 scope.go:117] "RemoveContainer" containerID="30b1f1ee2470c994dd773fbf0064111e7e2edd75a434055272341e29dfdeb010" Jul 13 22:07:22.714599 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:22.714572 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:07:22.714665 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:22.714643 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:07:22.714706 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:22.714681 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:07:23.708951 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:23.708915 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" event={"ID":"f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3","Type":"ContainerStarted","Data":"a0a6150dfb65f28e4cdc73105389a0c0bd594c063add58ee37ecf18f9257fad4"} Jul 13 22:07:23.877037 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:23.877002 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:07:23.878406 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:23.878368 2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cqx9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 13 22:07:23.878569 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:23.878428 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" podUID="f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 13 22:07:24.695599 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:24.695560 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-75cffb6f44-qnk2n" Jul 13 22:07:24.877375 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:24.877336 2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cqx9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 13 22:07:24.877820 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:24.877419 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" podUID="f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 13 22:07:25.877199 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:25.877162 2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cqx9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 13 22:07:25.877384 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:25.877225 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" podUID="f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 13 22:07:26.257955 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:26.257912 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:07:26.876851 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:26.876813 2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cqx9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 13 22:07:26.877026 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:26.876879 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" podUID="f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 13 22:07:27.881934 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:27.881904 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:07:27.883049 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:07:27.883027 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cqx9z" Jul 13 22:08:07.818070 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:07.818045 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 22:08:07.820405 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:07.820385 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 22:08:07.821380 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:07.821362 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 22:08:07.823284 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:07.823263 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 22:08:18.743676 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:18.743632 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm"] Jul 13 22:08:18.747009 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:18.746987 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:18.749871 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:18.749847 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 13 22:08:18.749983 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:18.749905 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 13 22:08:18.750896 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:18.750875 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 13 22:08:18.751000 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:18.750886 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2h8t7\"" Jul 13 22:08:18.756078 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:18.756057 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm"] Jul 13 22:08:18.905451 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:18.905417 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58bd174-0332-49e3-9b10-51085954d4b8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gvtqm\" (UID: \"f58bd174-0332-49e3-9b10-51085954d4b8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:18.905638 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:18.905523 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58bd174-0332-49e3-9b10-51085954d4b8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gvtqm\" (UID: \"f58bd174-0332-49e3-9b10-51085954d4b8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:18.905638 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:18.905591 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58bd174-0332-49e3-9b10-51085954d4b8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gvtqm\" (UID: \"f58bd174-0332-49e3-9b10-51085954d4b8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:18.905758 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:18.905648 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58bd174-0332-49e3-9b10-51085954d4b8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gvtqm\" (UID: \"f58bd174-0332-49e3-9b10-51085954d4b8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:18.905758 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:18.905737 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f58bd174-0332-49e3-9b10-51085954d4b8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gvtqm\" (UID: \"f58bd174-0332-49e3-9b10-51085954d4b8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:18.905869 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:18.905823 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt25c\" (UniqueName: \"kubernetes.io/projected/f58bd174-0332-49e3-9b10-51085954d4b8-kube-api-access-bt25c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gvtqm\" (UID: \"f58bd174-0332-49e3-9b10-51085954d4b8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:19.006788 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:19.006705 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58bd174-0332-49e3-9b10-51085954d4b8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gvtqm\" (UID: \"f58bd174-0332-49e3-9b10-51085954d4b8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:19.006788 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:19.006776 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58bd174-0332-49e3-9b10-51085954d4b8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gvtqm\" (UID: \"f58bd174-0332-49e3-9b10-51085954d4b8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:19.007001 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:19.006804 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58bd174-0332-49e3-9b10-51085954d4b8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gvtqm\" (UID: \"f58bd174-0332-49e3-9b10-51085954d4b8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:19.007001 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:19.006857 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58bd174-0332-49e3-9b10-51085954d4b8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gvtqm\" (UID: \"f58bd174-0332-49e3-9b10-51085954d4b8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:19.007001 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:19.006882 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f58bd174-0332-49e3-9b10-51085954d4b8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gvtqm\" (UID: \"f58bd174-0332-49e3-9b10-51085954d4b8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:19.007001 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:19.006979 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt25c\" (UniqueName: \"kubernetes.io/projected/f58bd174-0332-49e3-9b10-51085954d4b8-kube-api-access-bt25c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gvtqm\" (UID: \"f58bd174-0332-49e3-9b10-51085954d4b8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:19.007241 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:19.007214 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58bd174-0332-49e3-9b10-51085954d4b8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gvtqm\" (UID: \"f58bd174-0332-49e3-9b10-51085954d4b8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:19.007241 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:19.007235 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58bd174-0332-49e3-9b10-51085954d4b8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gvtqm\" (UID: \"f58bd174-0332-49e3-9b10-51085954d4b8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:19.007436 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:19.007414 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f58bd174-0332-49e3-9b10-51085954d4b8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gvtqm\" (UID: \"f58bd174-0332-49e3-9b10-51085954d4b8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:19.009028 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:19.009008 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58bd174-0332-49e3-9b10-51085954d4b8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gvtqm\" (UID: \"f58bd174-0332-49e3-9b10-51085954d4b8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:19.009194 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:19.009176 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58bd174-0332-49e3-9b10-51085954d4b8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gvtqm\" (UID: \"f58bd174-0332-49e3-9b10-51085954d4b8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:19.015187 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:19.015168 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt25c\" (UniqueName: \"kubernetes.io/projected/f58bd174-0332-49e3-9b10-51085954d4b8-kube-api-access-bt25c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gvtqm\" (UID: \"f58bd174-0332-49e3-9b10-51085954d4b8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:19.058218 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:19.058194 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:19.193127 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:19.193098 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm"] Jul 13 22:08:19.195230 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:08:19.195201 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58bd174_0332_49e3_9b10_51085954d4b8.slice/crio-806b0d380cb75c6b213f701f2dd49ab3b079b16862a04d238fd666c3abc98f08 WatchSource:0}: Error finding container 806b0d380cb75c6b213f701f2dd49ab3b079b16862a04d238fd666c3abc98f08: Status 404 returned error can't find the container with id 806b0d380cb75c6b213f701f2dd49ab3b079b16862a04d238fd666c3abc98f08 Jul 13 22:08:19.987774 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:19.987726 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" event={"ID":"f58bd174-0332-49e3-9b10-51085954d4b8","Type":"ContainerStarted","Data":"806b0d380cb75c6b213f701f2dd49ab3b079b16862a04d238fd666c3abc98f08"} Jul 13 22:08:20.994722 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:20.994688 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" event={"ID":"f58bd174-0332-49e3-9b10-51085954d4b8","Type":"ContainerStarted","Data":"1d578789672fbce3e1e2b1e8acf92873b1b73d91e0e0d92d25d97e27698fd046"} Jul 13 22:08:21.013321 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:21.013267 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" podStartSLOduration=1.823151718 podStartE2EDuration="3.013252526s" totalImagesPullingTime="1.190100808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:08:21.011562986 +0000 UTC m=+613.668767753" watchObservedRunningTime="2026-07-13 22:08:21.013252526 +0000 UTC m=+613.670457293" Jul 13 22:08:29.058679 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:29.058646 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:29.059125 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:29.058697 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:29.071127 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:29.071104 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:08:30.053696 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:08:30.053666 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gvtqm" Jul 13 22:09:41.609208 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:09:41.609175 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d47576bc5-fvrcx"] Jul 13 22:09:41.609799 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:09:41.609338 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6d47576bc5-fvrcx" podUID="e0a4ec79-e40c-4ed5-88c0-d46d721add18" containerName="authorino" containerID="cri-o://455ffc282a3407347e58b0bcf01a4aef6f6c975ec98c3570bac87549e1b8abd9" gracePeriod=30 Jul 13 22:09:41.873308 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:09:41.873250 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d47576bc5-fvrcx" Jul 13 22:09:41.916600 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:09:41.916573 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzw5v\" (UniqueName: \"kubernetes.io/projected/e0a4ec79-e40c-4ed5-88c0-d46d721add18-kube-api-access-fzw5v\") pod \"e0a4ec79-e40c-4ed5-88c0-d46d721add18\" (UID: \"e0a4ec79-e40c-4ed5-88c0-d46d721add18\") " Jul 13 22:09:41.916763 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:09:41.916622 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e0a4ec79-e40c-4ed5-88c0-d46d721add18-tls-cert\") pod \"e0a4ec79-e40c-4ed5-88c0-d46d721add18\" (UID: \"e0a4ec79-e40c-4ed5-88c0-d46d721add18\") " Jul 13 22:09:41.918771 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:09:41.918734 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a4ec79-e40c-4ed5-88c0-d46d721add18-kube-api-access-fzw5v" (OuterVolumeSpecName: "kube-api-access-fzw5v") pod "e0a4ec79-e40c-4ed5-88c0-d46d721add18" (UID: "e0a4ec79-e40c-4ed5-88c0-d46d721add18"). InnerVolumeSpecName "kube-api-access-fzw5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:09:41.927331 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:09:41.927308 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0a4ec79-e40c-4ed5-88c0-d46d721add18-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e0a4ec79-e40c-4ed5-88c0-d46d721add18" (UID: "e0a4ec79-e40c-4ed5-88c0-d46d721add18"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:09:42.017469 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:09:42.017263 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzw5v\" (UniqueName: \"kubernetes.io/projected/e0a4ec79-e40c-4ed5-88c0-d46d721add18-kube-api-access-fzw5v\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:09:42.017469 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:09:42.017294 2569 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e0a4ec79-e40c-4ed5-88c0-d46d721add18-tls-cert\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:09:42.407202 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:09:42.407159 2569 generic.go:356] "Generic (PLEG): container finished" podID="e0a4ec79-e40c-4ed5-88c0-d46d721add18" containerID="455ffc282a3407347e58b0bcf01a4aef6f6c975ec98c3570bac87549e1b8abd9" exitCode=0 Jul 13 22:09:42.407396 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:09:42.407268 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d47576bc5-fvrcx" event={"ID":"e0a4ec79-e40c-4ed5-88c0-d46d721add18","Type":"ContainerDied","Data":"455ffc282a3407347e58b0bcf01a4aef6f6c975ec98c3570bac87549e1b8abd9"} Jul 13 22:09:42.407396 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:09:42.407310 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d47576bc5-fvrcx" event={"ID":"e0a4ec79-e40c-4ed5-88c0-d46d721add18","Type":"ContainerDied","Data":"8b3c2a07a591d8532edc764b6fec06c1b0ff393a63a8576a13f7fa4761068dd6"} Jul 13 22:09:42.407396 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:09:42.407345 2569 scope.go:117] "RemoveContainer" containerID="455ffc282a3407347e58b0bcf01a4aef6f6c975ec98c3570bac87549e1b8abd9" Jul 13 22:09:42.407600 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:09:42.407584 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d47576bc5-fvrcx" Jul 13 22:09:42.421074 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:09:42.421044 2569 scope.go:117] "RemoveContainer" containerID="455ffc282a3407347e58b0bcf01a4aef6f6c975ec98c3570bac87549e1b8abd9" Jul 13 22:09:42.421705 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:09:42.421669 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"455ffc282a3407347e58b0bcf01a4aef6f6c975ec98c3570bac87549e1b8abd9\": container with ID starting with 455ffc282a3407347e58b0bcf01a4aef6f6c975ec98c3570bac87549e1b8abd9 not found: ID does not exist" containerID="455ffc282a3407347e58b0bcf01a4aef6f6c975ec98c3570bac87549e1b8abd9" Jul 13 22:09:42.421797 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:09:42.421729 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"455ffc282a3407347e58b0bcf01a4aef6f6c975ec98c3570bac87549e1b8abd9"} err="failed to get container status \"455ffc282a3407347e58b0bcf01a4aef6f6c975ec98c3570bac87549e1b8abd9\": rpc error: code = NotFound desc = could not find container \"455ffc282a3407347e58b0bcf01a4aef6f6c975ec98c3570bac87549e1b8abd9\": container with ID starting with 455ffc282a3407347e58b0bcf01a4aef6f6c975ec98c3570bac87549e1b8abd9 not found: ID does not exist" Jul 13 22:09:42.433084 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:09:42.433062 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d47576bc5-fvrcx"] Jul 13 22:09:42.444256 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:09:42.444235 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d47576bc5-fvrcx"] Jul 13 22:09:43.931341 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:09:43.931311 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0a4ec79-e40c-4ed5-88c0-d46d721add18" path="/var/lib/kubelet/pods/e0a4ec79-e40c-4ed5-88c0-d46d721add18/volumes" Jul 13 22:13:07.848926 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:13:07.848832 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 22:13:07.852599 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:13:07.852573 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 22:13:07.853392 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:13:07.853371 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 22:13:07.856372 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:13:07.856354 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 22:15:00.146239 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:00.146163 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733015-cc8c8"] Jul 13 22:15:00.146709 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:00.146565 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0a4ec79-e40c-4ed5-88c0-d46d721add18" containerName="authorino" Jul 13 22:15:00.146709 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:00.146577 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0a4ec79-e40c-4ed5-88c0-d46d721add18" containerName="authorino" Jul 13 22:15:00.146709 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:00.146639 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0a4ec79-e40c-4ed5-88c0-d46d721add18" containerName="authorino" Jul 13 22:15:00.149204 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:00.149187 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-cc8c8" Jul 13 22:15:00.151831 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:00.151812 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxd86\"" Jul 13 22:15:00.158758 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:00.158742 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-cc8c8" Jul 13 22:15:00.166761 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:00.166732 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733015-cc8c8"] Jul 13 22:15:00.303572 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:00.300083 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733015-cc8c8"] Jul 13 22:15:00.306232 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:00.304637 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 22:15:01.022059 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:01.022023 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-cc8c8" event={"ID":"d0fec174-463b-4051-8352-4039220fc9d3","Type":"ContainerStarted","Data":"941032e59433e0cb063196d13375da9ee04ee5c37c2eca96049c0663403c4225"} Jul 13 22:15:03.034026 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:03.033989 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-cc8c8" event={"ID":"d0fec174-463b-4051-8352-4039220fc9d3","Type":"ContainerStarted","Data":"c43c00285cc3974077c8614116418acedcc024c8dff1e2cd12449f0b32a6ee11"} Jul 13 22:15:03.049698 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:03.049649 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-cc8c8" podStartSLOduration=0.846681556 podStartE2EDuration="3.049634546s" totalImagesPullingTime="2.20295299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:15:03.048997813 +0000 UTC m=+1015.706202581" watchObservedRunningTime="2026-07-13 22:15:03.049634546 +0000 UTC m=+1015.706839314" Jul 13 22:15:38.937853 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:38.937816 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr"] Jul 13 22:15:38.938378 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:38.938000 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr" podUID="4ea9c003-5e7e-45b2-8f80-ca1b81068f89" containerName="manager" containerID="cri-o://689d4097bfc4bb007739f52093fc7abc367c6cf8219ad510fd9ed1cb7d5d6291" gracePeriod=10 Jul 13 22:15:42.230481 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:42.230449 2569 generic.go:356] "Generic (PLEG): container finished" podID="4ea9c003-5e7e-45b2-8f80-ca1b81068f89" containerID="689d4097bfc4bb007739f52093fc7abc367c6cf8219ad510fd9ed1cb7d5d6291" exitCode=0 Jul 13 22:15:42.230840 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:42.230525 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr" event={"ID":"4ea9c003-5e7e-45b2-8f80-ca1b81068f89","Type":"ContainerDied","Data":"689d4097bfc4bb007739f52093fc7abc367c6cf8219ad510fd9ed1cb7d5d6291"} Jul 13 22:15:42.693327 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:42.693304 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr" Jul 13 22:15:42.773160 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:42.773132 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ea9c003-5e7e-45b2-8f80-ca1b81068f89-extensions-socket-volume\") pod \"4ea9c003-5e7e-45b2-8f80-ca1b81068f89\" (UID: \"4ea9c003-5e7e-45b2-8f80-ca1b81068f89\") " Jul 13 22:15:42.773322 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:42.773221 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgbd5\" (UniqueName: \"kubernetes.io/projected/4ea9c003-5e7e-45b2-8f80-ca1b81068f89-kube-api-access-zgbd5\") pod \"4ea9c003-5e7e-45b2-8f80-ca1b81068f89\" (UID: \"4ea9c003-5e7e-45b2-8f80-ca1b81068f89\") " Jul 13 22:15:42.773501 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:42.773478 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea9c003-5e7e-45b2-8f80-ca1b81068f89-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4ea9c003-5e7e-45b2-8f80-ca1b81068f89" (UID: "4ea9c003-5e7e-45b2-8f80-ca1b81068f89"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:15:42.775081 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:42.775054 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea9c003-5e7e-45b2-8f80-ca1b81068f89-kube-api-access-zgbd5" (OuterVolumeSpecName: "kube-api-access-zgbd5") pod "4ea9c003-5e7e-45b2-8f80-ca1b81068f89" (UID: "4ea9c003-5e7e-45b2-8f80-ca1b81068f89"). InnerVolumeSpecName "kube-api-access-zgbd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:15:42.874783 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:42.874723 2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ea9c003-5e7e-45b2-8f80-ca1b81068f89-extensions-socket-volume\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:15:42.874783 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:42.874746 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgbd5\" (UniqueName: \"kubernetes.io/projected/4ea9c003-5e7e-45b2-8f80-ca1b81068f89-kube-api-access-zgbd5\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:15:43.236644 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:43.236610 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr" event={"ID":"4ea9c003-5e7e-45b2-8f80-ca1b81068f89","Type":"ContainerDied","Data":"ed6b7140048d77ddb1da966a8f6c1e6233657c7994d3f2bcd79e6e71b6cc9a25"} Jul 13 22:15:43.237087 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:43.236668 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr" Jul 13 22:15:43.237087 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:43.236683 2569 scope.go:117] "RemoveContainer" containerID="689d4097bfc4bb007739f52093fc7abc367c6cf8219ad510fd9ed1cb7d5d6291" Jul 13 22:15:43.258779 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:43.258755 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr"] Jul 13 22:15:43.262821 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:43.262801 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzkxr"] Jul 13 22:15:43.931287 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:15:43.931252 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ea9c003-5e7e-45b2-8f80-ca1b81068f89" path="/var/lib/kubelet/pods/4ea9c003-5e7e-45b2-8f80-ca1b81068f89/volumes" Jul 13 22:16:00.011303 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:00.011267 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733015-cc8c8"] Jul 13 22:16:00.011812 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:00.011461 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-cc8c8" podUID="d0fec174-463b-4051-8352-4039220fc9d3" containerName="cleanup" containerID="cri-o://c43c00285cc3974077c8614116418acedcc024c8dff1e2cd12449f0b32a6ee11" gracePeriod=30 Jul 13 22:16:30.202264 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:30.202235 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-cc8c8" Jul 13 22:16:30.475827 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:30.475799 2569 generic.go:356] "Generic (PLEG): container finished" podID="d0fec174-463b-4051-8352-4039220fc9d3" containerID="c43c00285cc3974077c8614116418acedcc024c8dff1e2cd12449f0b32a6ee11" exitCode=137 Jul 13 22:16:30.475996 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:30.475839 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-cc8c8" Jul 13 22:16:30.475996 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:30.475888 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-cc8c8" event={"ID":"d0fec174-463b-4051-8352-4039220fc9d3","Type":"ContainerDied","Data":"c43c00285cc3974077c8614116418acedcc024c8dff1e2cd12449f0b32a6ee11"} Jul 13 22:16:30.475996 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:30.475939 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-cc8c8" event={"ID":"d0fec174-463b-4051-8352-4039220fc9d3","Type":"ContainerDied","Data":"941032e59433e0cb063196d13375da9ee04ee5c37c2eca96049c0663403c4225"} Jul 13 22:16:30.475996 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:30.475963 2569 scope.go:117] "RemoveContainer" containerID="c43c00285cc3974077c8614116418acedcc024c8dff1e2cd12449f0b32a6ee11" Jul 13 22:16:30.484728 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:30.484712 2569 scope.go:117] "RemoveContainer" containerID="c43c00285cc3974077c8614116418acedcc024c8dff1e2cd12449f0b32a6ee11" Jul 13 22:16:30.484966 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:16:30.484943 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c43c00285cc3974077c8614116418acedcc024c8dff1e2cd12449f0b32a6ee11\": container with ID starting with c43c00285cc3974077c8614116418acedcc024c8dff1e2cd12449f0b32a6ee11 not found: ID does not exist" containerID="c43c00285cc3974077c8614116418acedcc024c8dff1e2cd12449f0b32a6ee11" Jul 13 22:16:30.485033 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:30.484976 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c43c00285cc3974077c8614116418acedcc024c8dff1e2cd12449f0b32a6ee11"} err="failed to get container status \"c43c00285cc3974077c8614116418acedcc024c8dff1e2cd12449f0b32a6ee11\": rpc error: code = NotFound desc = could not find container \"c43c00285cc3974077c8614116418acedcc024c8dff1e2cd12449f0b32a6ee11\": container with ID starting with c43c00285cc3974077c8614116418acedcc024c8dff1e2cd12449f0b32a6ee11 not found: ID does not exist" Jul 13 22:16:30.496831 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:30.496811 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733015-cc8c8"] Jul 13 22:16:30.502280 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:30.502260 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733015-cc8c8"] Jul 13 22:16:31.930959 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:31.930924 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0fec174-463b-4051-8352-4039220fc9d3" path="/var/lib/kubelet/pods/d0fec174-463b-4051-8352-4039220fc9d3/volumes" Jul 13 22:16:40.061221 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:40.061180 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9bbrq"] Jul 13 22:16:40.061781 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:40.061761 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0fec174-463b-4051-8352-4039220fc9d3" containerName="cleanup" Jul 13 22:16:40.061868 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:40.061784 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0fec174-463b-4051-8352-4039220fc9d3" containerName="cleanup" Jul 13 22:16:40.061868 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:40.061816 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea9c003-5e7e-45b2-8f80-ca1b81068f89" containerName="manager" Jul 13 22:16:40.061868 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:40.061826 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea9c003-5e7e-45b2-8f80-ca1b81068f89" containerName="manager" Jul 13 22:16:40.062022 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:40.061945 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0fec174-463b-4051-8352-4039220fc9d3" containerName="cleanup" Jul 13 22:16:40.062022 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:40.061958 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea9c003-5e7e-45b2-8f80-ca1b81068f89" containerName="manager" Jul 13 22:16:40.064956 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:40.064935 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9bbrq" Jul 13 22:16:40.067754 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:40.067733 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q69hx\"" Jul 13 22:16:40.078341 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:40.078320 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9bbrq"] Jul 13 22:16:40.234681 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:40.234648 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwtrd\" (UniqueName: \"kubernetes.io/projected/cdda6f89-cd2e-488b-8ebf-26974650d601-kube-api-access-wwtrd\") pod \"kuadrant-operator-controller-manager-86878c769f-9bbrq\" (UID: \"cdda6f89-cd2e-488b-8ebf-26974650d601\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9bbrq" Jul 13 22:16:40.234874 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:40.234735 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cdda6f89-cd2e-488b-8ebf-26974650d601-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9bbrq\" (UID: \"cdda6f89-cd2e-488b-8ebf-26974650d601\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9bbrq" Jul 13 22:16:40.335475 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:40.335403 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cdda6f89-cd2e-488b-8ebf-26974650d601-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9bbrq\" (UID: \"cdda6f89-cd2e-488b-8ebf-26974650d601\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9bbrq" Jul 13 22:16:40.335475 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:40.335453 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwtrd\" (UniqueName: \"kubernetes.io/projected/cdda6f89-cd2e-488b-8ebf-26974650d601-kube-api-access-wwtrd\") pod \"kuadrant-operator-controller-manager-86878c769f-9bbrq\" (UID: \"cdda6f89-cd2e-488b-8ebf-26974650d601\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9bbrq" Jul 13 22:16:40.335791 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:40.335770 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cdda6f89-cd2e-488b-8ebf-26974650d601-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9bbrq\" (UID: \"cdda6f89-cd2e-488b-8ebf-26974650d601\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9bbrq" Jul 13 22:16:40.345184 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:40.345162 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwtrd\" (UniqueName: \"kubernetes.io/projected/cdda6f89-cd2e-488b-8ebf-26974650d601-kube-api-access-wwtrd\") pod \"kuadrant-operator-controller-manager-86878c769f-9bbrq\" (UID: \"cdda6f89-cd2e-488b-8ebf-26974650d601\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9bbrq" Jul 13 22:16:40.375847 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:40.375825 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9bbrq" Jul 13 22:16:40.513723 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:40.513694 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9bbrq"] Jul 13 22:16:40.515670 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:16:40.515646 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdda6f89_cd2e_488b_8ebf_26974650d601.slice/crio-c69206cafc9ee68a8a9b7c0633977e396ed8e2efc79feed69adf316db4061636 WatchSource:0}: Error finding container c69206cafc9ee68a8a9b7c0633977e396ed8e2efc79feed69adf316db4061636: Status 404 returned error can't find the container with id c69206cafc9ee68a8a9b7c0633977e396ed8e2efc79feed69adf316db4061636 Jul 13 22:16:40.529684 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:40.529658 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9bbrq" event={"ID":"cdda6f89-cd2e-488b-8ebf-26974650d601","Type":"ContainerStarted","Data":"c69206cafc9ee68a8a9b7c0633977e396ed8e2efc79feed69adf316db4061636"} Jul 13 22:16:41.537074 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:41.537042 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9bbrq" event={"ID":"cdda6f89-cd2e-488b-8ebf-26974650d601","Type":"ContainerStarted","Data":"b7623f8973e7e221e02b5280664951630090b52553c2b3197da2e727f226b40e"} Jul 13 22:16:41.537462 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:41.537176 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9bbrq" Jul 13 22:16:41.571797 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:41.571747 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9bbrq" podStartSLOduration=1.5717319509999998 podStartE2EDuration="1.571731951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:16:41.561054204 +0000 UTC m=+1114.218259095" watchObservedRunningTime="2026-07-13 22:16:41.571731951 +0000 UTC m=+1114.228936718" Jul 13 22:16:52.545409 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:16:52.545374 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9bbrq" Jul 13 22:18:07.878309 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:18:07.878207 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 22:18:07.882484 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:18:07.881487 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 22:18:07.885718 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:18:07.885699 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 22:18:07.888817 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:18:07.888799 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 22:22:57.519193 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.519163 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk"] Jul 13 22:22:57.522379 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.522360 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.525001 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.524983 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c3e759fc-openshift-default-dockercfg-l4qrs\"" Jul 13 22:22:57.547254 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.547226 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk"] Jul 13 22:22:57.551152 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.551123 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-istio-data\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.551243 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.551181 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-credential-socket\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.551243 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.551225 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-workload-certs\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.551368 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.551344 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2407c086-9077-40bf-9663-6f43e75e90cc-istio-podinfo\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.551431 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.551402 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2407c086-9077-40bf-9663-6f43e75e90cc-istio-token\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.551489 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.551471 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-istio-envoy\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.551573 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.551508 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-workload-socket\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.551641 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.551585 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2407c086-9077-40bf-9663-6f43e75e90cc-istiod-ca-cert\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.551641 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.551612 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdh6b\" (UniqueName: \"kubernetes.io/projected/2407c086-9077-40bf-9663-6f43e75e90cc-kube-api-access-xdh6b\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.652392 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.652356 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2407c086-9077-40bf-9663-6f43e75e90cc-istiod-ca-cert\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.652392 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.652389 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdh6b\" (UniqueName: \"kubernetes.io/projected/2407c086-9077-40bf-9663-6f43e75e90cc-kube-api-access-xdh6b\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.652607 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.652518 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-istio-data\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.652646 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.652605 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-credential-socket\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.652646 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.652625 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-workload-certs\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.652737 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.652658 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2407c086-9077-40bf-9663-6f43e75e90cc-istio-podinfo\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.652737 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.652712 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2407c086-9077-40bf-9663-6f43e75e90cc-istio-token\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.652842 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.652739 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-istio-envoy\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.652928 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.652907 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-istio-data\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.652994 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.652959 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-workload-socket\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.653050 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.653000 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-workload-certs\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.653050 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.653023 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-credential-socket\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.653127 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.653106 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2407c086-9077-40bf-9663-6f43e75e90cc-istiod-ca-cert\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.653252 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.653235 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-workload-socket\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.654965 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.654944 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-istio-envoy\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.655198 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.655177 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2407c086-9077-40bf-9663-6f43e75e90cc-istio-podinfo\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.660851 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.660824 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2407c086-9077-40bf-9663-6f43e75e90cc-istio-token\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.660990 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.660973 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdh6b\" (UniqueName: \"kubernetes.io/projected/2407c086-9077-40bf-9663-6f43e75e90cc-kube-api-access-xdh6b\") pod \"e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.833050 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.832979 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:57.975002 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.974978 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk"] Jul 13 22:22:57.976799 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:22:57.976771 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2407c086_9077_40bf_9663_6f43e75e90cc.slice/crio-4ee1269bd343231f0f700a26ece1969e523e68fbf9f1e1a4f1ba90dab92e4b1f WatchSource:0}: Error finding container 4ee1269bd343231f0f700a26ece1969e523e68fbf9f1e1a4f1ba90dab92e4b1f: Status 404 returned error can't find the container with id 4ee1269bd343231f0f700a26ece1969e523e68fbf9f1e1a4f1ba90dab92e4b1f Jul 13 22:22:57.978972 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.978940 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:22:57.979085 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.979001 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:22:57.979085 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:57.979028 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:22:58.434804 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:58.434766 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" event={"ID":"2407c086-9077-40bf-9663-6f43e75e90cc","Type":"ContainerStarted","Data":"aeafbd339c23f1ea2aa831d2e9b98217bec80519e75cd32e18aeea708f82dd04"} Jul 13 22:22:58.434976 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:58.434859 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" event={"ID":"2407c086-9077-40bf-9663-6f43e75e90cc","Type":"ContainerStarted","Data":"4ee1269bd343231f0f700a26ece1969e523e68fbf9f1e1a4f1ba90dab92e4b1f"} Jul 13 22:22:58.457421 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:58.457360 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" podStartSLOduration=1.457341192 podStartE2EDuration="1.457341192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:22:58.456596171 +0000 UTC m=+1491.113800938" watchObservedRunningTime="2026-07-13 22:22:58.457341192 +0000 UTC m=+1491.114545963" Jul 13 22:22:58.831206 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:58.831173 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx"] Jul 13 22:22:58.835019 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:58.834995 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:58.835159 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:58.835097 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" Jul 13 22:22:58.838087 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:58.838064 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c3e759fc-serving-cert\"" Jul 13 22:22:58.852230 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:58.852205 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx"] Jul 13 22:22:58.863716 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:58.863694 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2201234-4196-439e-82ec-355922d77959-openshift-service-ca\") pod \"maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx\" (UID: \"b2201234-4196-439e-82ec-355922d77959\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" Jul 13 22:22:58.863823 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:58.863726 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp8fc\" (UniqueName: \"kubernetes.io/projected/b2201234-4196-439e-82ec-355922d77959-kube-api-access-bp8fc\") pod \"maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx\" (UID: \"b2201234-4196-439e-82ec-355922d77959\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" Jul 13 22:22:58.863823 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:58.863774 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2201234-4196-439e-82ec-355922d77959-maas-api-tls\") pod \"maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx\" (UID: \"b2201234-4196-439e-82ec-355922d77959\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" Jul 13 22:22:58.965023 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:58.964989 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2201234-4196-439e-82ec-355922d77959-openshift-service-ca\") pod \"maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx\" (UID: \"b2201234-4196-439e-82ec-355922d77959\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" Jul 13 22:22:58.965217 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:58.965029 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp8fc\" (UniqueName: \"kubernetes.io/projected/b2201234-4196-439e-82ec-355922d77959-kube-api-access-bp8fc\") pod \"maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx\" (UID: \"b2201234-4196-439e-82ec-355922d77959\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" Jul 13 22:22:58.965217 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:58.965089 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2201234-4196-439e-82ec-355922d77959-maas-api-tls\") pod \"maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx\" (UID: \"b2201234-4196-439e-82ec-355922d77959\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" Jul 13 22:22:58.965217 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:22:58.965194 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-serving-cert: secret "maas-api-e2e-ait-c3e759fc-serving-cert" not found Jul 13 22:22:58.965390 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:22:58.965259 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2201234-4196-439e-82ec-355922d77959-maas-api-tls podName:b2201234-4196-439e-82ec-355922d77959 nodeName:}" failed. No retries permitted until 2026-07-13 22:22:59.465244256 +0000 UTC m=+1492.122449002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b2201234-4196-439e-82ec-355922d77959-maas-api-tls") pod "maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" (UID: "b2201234-4196-439e-82ec-355922d77959") : secret "maas-api-e2e-ait-c3e759fc-serving-cert" not found Jul 13 22:22:58.965655 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:58.965634 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2201234-4196-439e-82ec-355922d77959-openshift-service-ca\") pod \"maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx\" (UID: \"b2201234-4196-439e-82ec-355922d77959\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" Jul 13 22:22:58.974051 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:58.974029 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp8fc\" (UniqueName: \"kubernetes.io/projected/b2201234-4196-439e-82ec-355922d77959-kube-api-access-bp8fc\") pod \"maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx\" (UID: \"b2201234-4196-439e-82ec-355922d77959\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" Jul 13 22:22:59.469942 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:59.469906 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2201234-4196-439e-82ec-355922d77959-maas-api-tls\") pod \"maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx\" (UID: \"b2201234-4196-439e-82ec-355922d77959\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" Jul 13 22:22:59.472260 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:59.472237 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2201234-4196-439e-82ec-355922d77959-maas-api-tls\") pod \"maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx\" (UID: \"b2201234-4196-439e-82ec-355922d77959\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" Jul 13 22:22:59.745577 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:59.745472 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" Jul 13 22:22:59.839660 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:59.839633 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:22:59.886532 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:59.886504 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx"] Jul 13 22:22:59.889334 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:22:59.889302 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2201234_4196_439e_82ec_355922d77959.slice/crio-e10e795abf0456e0ad7fe71dcc735bd3774765a31e8152777d3757fcbc773b91 WatchSource:0}: Error finding container e10e795abf0456e0ad7fe71dcc735bd3774765a31e8152777d3757fcbc773b91: Status 404 returned error can't find the container with id e10e795abf0456e0ad7fe71dcc735bd3774765a31e8152777d3757fcbc773b91 Jul 13 22:22:59.890796 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:22:59.890777 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 22:23:00.445831 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:00.445792 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" event={"ID":"b2201234-4196-439e-82ec-355922d77959","Type":"ContainerStarted","Data":"e10e795abf0456e0ad7fe71dcc735bd3774765a31e8152777d3757fcbc773b91"} Jul 13 22:23:00.445984 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:00.445947 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:23:00.446928 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:00.446910 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:23:01.453051 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:01.453013 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" event={"ID":"b2201234-4196-439e-82ec-355922d77959","Type":"ContainerStarted","Data":"eb8c0edb11dbd4d81586c8e943219e0f50a70eebcc9a7861de6661caa8487285"} Jul 13 22:23:01.473231 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:01.473184 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" podStartSLOduration=2.782070156 podStartE2EDuration="3.473167952s" totalImagesPullingTime="691.097796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:23:01.471091766 +0000 UTC m=+1494.128296534" watchObservedRunningTime="2026-07-13 22:23:01.473167952 +0000 UTC m=+1494.130372720" Jul 13 22:23:02.458701 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:02.458668 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" Jul 13 22:23:03.930605 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:03.930574 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk"] Jul 13 22:23:03.931087 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:03.930733 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" podUID="2407c086-9077-40bf-9663-6f43e75e90cc" containerName="istio-proxy" containerID="cri-o://aeafbd339c23f1ea2aa831d2e9b98217bec80519e75cd32e18aeea708f82dd04" gracePeriod=30 Jul 13 22:23:04.187562 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:04.187516 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx"] Jul 13 22:23:04.468454 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:04.468359 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" podUID="b2201234-4196-439e-82ec-355922d77959" containerName="maas-api" containerID="cri-o://eb8c0edb11dbd4d81586c8e943219e0f50a70eebcc9a7861de6661caa8487285" gracePeriod=30 Jul 13 22:23:04.766595 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:04.766530 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" Jul 13 22:23:04.822727 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:04.822692 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp8fc\" (UniqueName: \"kubernetes.io/projected/b2201234-4196-439e-82ec-355922d77959-kube-api-access-bp8fc\") pod \"b2201234-4196-439e-82ec-355922d77959\" (UID: \"b2201234-4196-439e-82ec-355922d77959\") " Jul 13 22:23:04.822896 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:04.822743 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2201234-4196-439e-82ec-355922d77959-maas-api-tls\") pod \"b2201234-4196-439e-82ec-355922d77959\" (UID: \"b2201234-4196-439e-82ec-355922d77959\") " Jul 13 22:23:04.822896 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:04.822854 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2201234-4196-439e-82ec-355922d77959-openshift-service-ca\") pod \"b2201234-4196-439e-82ec-355922d77959\" (UID: \"b2201234-4196-439e-82ec-355922d77959\") " Jul 13 22:23:04.823346 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:04.823321 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2201234-4196-439e-82ec-355922d77959-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b2201234-4196-439e-82ec-355922d77959" (UID: "b2201234-4196-439e-82ec-355922d77959"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:23:04.825314 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:04.825285 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2201234-4196-439e-82ec-355922d77959-kube-api-access-bp8fc" (OuterVolumeSpecName: "kube-api-access-bp8fc") pod "b2201234-4196-439e-82ec-355922d77959" (UID: "b2201234-4196-439e-82ec-355922d77959"). InnerVolumeSpecName "kube-api-access-bp8fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:23:04.825648 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:04.825619 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2201234-4196-439e-82ec-355922d77959-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b2201234-4196-439e-82ec-355922d77959" (UID: "b2201234-4196-439e-82ec-355922d77959"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:23:04.923427 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:04.923395 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2201234-4196-439e-82ec-355922d77959-openshift-service-ca\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:04.923427 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:04.923423 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp8fc\" (UniqueName: \"kubernetes.io/projected/b2201234-4196-439e-82ec-355922d77959-kube-api-access-bp8fc\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:04.923427 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:04.923434 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2201234-4196-439e-82ec-355922d77959-maas-api-tls\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:05.474104 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:05.474074 2569 generic.go:356] "Generic (PLEG): container finished" podID="b2201234-4196-439e-82ec-355922d77959" containerID="eb8c0edb11dbd4d81586c8e943219e0f50a70eebcc9a7861de6661caa8487285" exitCode=0 Jul 13 22:23:05.474593 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:05.474119 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" Jul 13 22:23:05.474593 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:05.474166 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" event={"ID":"b2201234-4196-439e-82ec-355922d77959","Type":"ContainerDied","Data":"eb8c0edb11dbd4d81586c8e943219e0f50a70eebcc9a7861de6661caa8487285"} Jul 13 22:23:05.474593 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:05.474231 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx" event={"ID":"b2201234-4196-439e-82ec-355922d77959","Type":"ContainerDied","Data":"e10e795abf0456e0ad7fe71dcc735bd3774765a31e8152777d3757fcbc773b91"} Jul 13 22:23:05.474593 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:05.474271 2569 scope.go:117] "RemoveContainer" containerID="eb8c0edb11dbd4d81586c8e943219e0f50a70eebcc9a7861de6661caa8487285" Jul 13 22:23:05.490247 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:05.490221 2569 scope.go:117] "RemoveContainer" containerID="eb8c0edb11dbd4d81586c8e943219e0f50a70eebcc9a7861de6661caa8487285" Jul 13 22:23:05.490720 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:23:05.490690 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb8c0edb11dbd4d81586c8e943219e0f50a70eebcc9a7861de6661caa8487285\": container with ID starting with eb8c0edb11dbd4d81586c8e943219e0f50a70eebcc9a7861de6661caa8487285 not found: ID does not exist" containerID="eb8c0edb11dbd4d81586c8e943219e0f50a70eebcc9a7861de6661caa8487285" Jul 13 22:23:05.490830 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:05.490730 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb8c0edb11dbd4d81586c8e943219e0f50a70eebcc9a7861de6661caa8487285"} err="failed to get container status \"eb8c0edb11dbd4d81586c8e943219e0f50a70eebcc9a7861de6661caa8487285\": rpc error: code = NotFound desc = could not find container \"eb8c0edb11dbd4d81586c8e943219e0f50a70eebcc9a7861de6661caa8487285\": container with ID starting with eb8c0edb11dbd4d81586c8e943219e0f50a70eebcc9a7861de6661caa8487285 not found: ID does not exist" Jul 13 22:23:05.505490 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:05.505455 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx"] Jul 13 22:23:05.510377 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:05.510358 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c3e759fc-58cc5c54d-znqjx"] Jul 13 22:23:05.931240 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:05.931152 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2201234-4196-439e-82ec-355922d77959" path="/var/lib/kubelet/pods/b2201234-4196-439e-82ec-355922d77959/volumes" Jul 13 22:23:07.914227 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:07.914116 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 22:23:07.935250 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:07.925856 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 22:23:07.935250 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:07.931347 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 22:23:07.935250 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:07.934998 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 22:23:09.974494 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:09.974460 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n"] Jul 13 22:23:09.975054 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:09.975031 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2201234-4196-439e-82ec-355922d77959" containerName="maas-api" Jul 13 22:23:09.975140 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:09.975060 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2201234-4196-439e-82ec-355922d77959" containerName="maas-api" Jul 13 22:23:09.975198 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:09.975187 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2201234-4196-439e-82ec-355922d77959" containerName="maas-api" Jul 13 22:23:09.978889 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:09.978862 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:09.982333 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:09.982311 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3114120a-openshift-default-dockercfg-fcwfh\"" Jul 13 22:23:09.996934 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:09.996904 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n"] Jul 13 22:23:10.069879 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.069846 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istiod-ca-cert\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.070041 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.069893 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-token\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.070041 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.069916 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-podinfo\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.070041 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.069998 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-credential-socket\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.070203 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.070045 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-workload-certs\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.070203 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.070095 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwwb2\" (UniqueName: \"kubernetes.io/projected/7d234f3e-a3e1-42f3-a599-3266de2d7a75-kube-api-access-nwwb2\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.070203 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.070150 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-data\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.070203 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.070183 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-envoy\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.070361 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.070241 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-workload-socket\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.170744 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.170714 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-token\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.171927 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.170752 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-podinfo\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.171927 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.170779 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-credential-socket\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.171927 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.170892 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-workload-certs\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.171927 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.170939 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwwb2\" (UniqueName: \"kubernetes.io/projected/7d234f3e-a3e1-42f3-a599-3266de2d7a75-kube-api-access-nwwb2\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.171927 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.171072 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-data\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.171927 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.171122 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-envoy\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.171927 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.171187 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-workload-socket\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.171927 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.171219 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-credential-socket\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.171927 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.171263 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-workload-certs\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.171927 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.171438 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-data\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.171927 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.171570 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istiod-ca-cert\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.171927 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.171874 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-workload-socket\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.172627 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.172102 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istiod-ca-cert\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.173175 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.173154 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-podinfo\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.173659 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.173641 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-envoy\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.181226 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.181207 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-token\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.181578 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.181558 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwwb2\" (UniqueName: \"kubernetes.io/projected/7d234f3e-a3e1-42f3-a599-3266de2d7a75-kube-api-access-nwwb2\") pod \"e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.295971 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.295940 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:10.389902 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.389873 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:23:10.450006 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.449978 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n"] Jul 13 22:23:10.451665 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:23:10.451638 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d234f3e_a3e1_42f3_a599_3266de2d7a75.slice/crio-211ef3f842da5843e57508a6f74555852a0f324b4a89e5152568ada808d07818 WatchSource:0}: Error finding container 211ef3f842da5843e57508a6f74555852a0f324b4a89e5152568ada808d07818: Status 404 returned error can't find the container with id 211ef3f842da5843e57508a6f74555852a0f324b4a89e5152568ada808d07818 Jul 13 22:23:10.453910 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.453871 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:23:10.454027 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.453954 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:23:10.454027 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.453993 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:23:10.474684 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.474663 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-istio-envoy\") pod \"2407c086-9077-40bf-9663-6f43e75e90cc\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " Jul 13 22:23:10.474786 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.474692 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdh6b\" (UniqueName: \"kubernetes.io/projected/2407c086-9077-40bf-9663-6f43e75e90cc-kube-api-access-xdh6b\") pod \"2407c086-9077-40bf-9663-6f43e75e90cc\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " Jul 13 22:23:10.474786 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.474729 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-istio-data\") pod \"2407c086-9077-40bf-9663-6f43e75e90cc\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " Jul 13 22:23:10.474786 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.474746 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2407c086-9077-40bf-9663-6f43e75e90cc-istio-token\") pod \"2407c086-9077-40bf-9663-6f43e75e90cc\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " Jul 13 22:23:10.474786 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.474764 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2407c086-9077-40bf-9663-6f43e75e90cc-istiod-ca-cert\") pod \"2407c086-9077-40bf-9663-6f43e75e90cc\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " Jul 13 22:23:10.475000 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.474838 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-credential-socket\") pod \"2407c086-9077-40bf-9663-6f43e75e90cc\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " Jul 13 22:23:10.475000 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.474875 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2407c086-9077-40bf-9663-6f43e75e90cc-istio-podinfo\") pod \"2407c086-9077-40bf-9663-6f43e75e90cc\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " Jul 13 22:23:10.475000 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.474917 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-workload-certs\") pod \"2407c086-9077-40bf-9663-6f43e75e90cc\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " Jul 13 22:23:10.475000 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.474979 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-workload-socket\") pod \"2407c086-9077-40bf-9663-6f43e75e90cc\" (UID: \"2407c086-9077-40bf-9663-6f43e75e90cc\") " Jul 13 22:23:10.475217 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.475110 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2407c086-9077-40bf-9663-6f43e75e90cc" (UID: "2407c086-9077-40bf-9663-6f43e75e90cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:23:10.475217 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.475160 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2407c086-9077-40bf-9663-6f43e75e90cc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2407c086-9077-40bf-9663-6f43e75e90cc" (UID: "2407c086-9077-40bf-9663-6f43e75e90cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:23:10.475322 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.475302 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2407c086-9077-40bf-9663-6f43e75e90cc" (UID: "2407c086-9077-40bf-9663-6f43e75e90cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:23:10.475430 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.475413 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-credential-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:10.475495 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.475435 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-workload-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:10.475495 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.475450 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2407c086-9077-40bf-9663-6f43e75e90cc-istiod-ca-cert\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:10.475495 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.475485 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2407c086-9077-40bf-9663-6f43e75e90cc" (UID: "2407c086-9077-40bf-9663-6f43e75e90cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:23:10.475672 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.475583 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-istio-data" (OuterVolumeSpecName: "istio-data") pod "2407c086-9077-40bf-9663-6f43e75e90cc" (UID: "2407c086-9077-40bf-9663-6f43e75e90cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:23:10.476965 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.476917 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2407c086-9077-40bf-9663-6f43e75e90cc-istio-token" (OuterVolumeSpecName: "istio-token") pod "2407c086-9077-40bf-9663-6f43e75e90cc" (UID: "2407c086-9077-40bf-9663-6f43e75e90cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:23:10.477042 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.476958 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2407c086-9077-40bf-9663-6f43e75e90cc-kube-api-access-xdh6b" (OuterVolumeSpecName: "kube-api-access-xdh6b") pod "2407c086-9077-40bf-9663-6f43e75e90cc" (UID: "2407c086-9077-40bf-9663-6f43e75e90cc"). InnerVolumeSpecName "kube-api-access-xdh6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:23:10.477042 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.476970 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2407c086-9077-40bf-9663-6f43e75e90cc" (UID: "2407c086-9077-40bf-9663-6f43e75e90cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:23:10.477176 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.477156 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2407c086-9077-40bf-9663-6f43e75e90cc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2407c086-9077-40bf-9663-6f43e75e90cc" (UID: "2407c086-9077-40bf-9663-6f43e75e90cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:23:10.504816 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.504789 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" event={"ID":"7d234f3e-a3e1-42f3-a599-3266de2d7a75","Type":"ContainerStarted","Data":"211ef3f842da5843e57508a6f74555852a0f324b4a89e5152568ada808d07818"} Jul 13 22:23:10.506178 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.506156 2569 generic.go:356] "Generic (PLEG): container finished" podID="2407c086-9077-40bf-9663-6f43e75e90cc" containerID="aeafbd339c23f1ea2aa831d2e9b98217bec80519e75cd32e18aeea708f82dd04" exitCode=0 Jul 13 22:23:10.506286 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.506186 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" event={"ID":"2407c086-9077-40bf-9663-6f43e75e90cc","Type":"ContainerDied","Data":"aeafbd339c23f1ea2aa831d2e9b98217bec80519e75cd32e18aeea708f82dd04"} Jul 13 22:23:10.506286 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.506205 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" Jul 13 22:23:10.506286 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.506223 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk" event={"ID":"2407c086-9077-40bf-9663-6f43e75e90cc","Type":"ContainerDied","Data":"4ee1269bd343231f0f700a26ece1969e523e68fbf9f1e1a4f1ba90dab92e4b1f"} Jul 13 22:23:10.506286 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.506246 2569 scope.go:117] "RemoveContainer" containerID="aeafbd339c23f1ea2aa831d2e9b98217bec80519e75cd32e18aeea708f82dd04" Jul 13 22:23:10.515738 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.515719 2569 scope.go:117] "RemoveContainer" containerID="aeafbd339c23f1ea2aa831d2e9b98217bec80519e75cd32e18aeea708f82dd04" Jul 13 22:23:10.515987 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:23:10.515968 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeafbd339c23f1ea2aa831d2e9b98217bec80519e75cd32e18aeea708f82dd04\": container with ID starting with aeafbd339c23f1ea2aa831d2e9b98217bec80519e75cd32e18aeea708f82dd04 not found: ID does not exist" containerID="aeafbd339c23f1ea2aa831d2e9b98217bec80519e75cd32e18aeea708f82dd04" Jul 13 22:23:10.516086 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.515993 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeafbd339c23f1ea2aa831d2e9b98217bec80519e75cd32e18aeea708f82dd04"} err="failed to get container status \"aeafbd339c23f1ea2aa831d2e9b98217bec80519e75cd32e18aeea708f82dd04\": rpc error: code = NotFound desc = could not find container \"aeafbd339c23f1ea2aa831d2e9b98217bec80519e75cd32e18aeea708f82dd04\": container with ID starting with aeafbd339c23f1ea2aa831d2e9b98217bec80519e75cd32e18aeea708f82dd04 not found: ID does not exist" Jul 13 22:23:10.532890 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.532869 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk"] Jul 13 22:23:10.538101 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.538074 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c3e759fc-openshift-default-655f8f4bf-rnmzk"] Jul 13 22:23:10.576260 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.576231 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-istio-envoy\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:10.576260 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.576260 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdh6b\" (UniqueName: \"kubernetes.io/projected/2407c086-9077-40bf-9663-6f43e75e90cc-kube-api-access-xdh6b\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:10.576453 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.576271 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-istio-data\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:10.576453 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.576280 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2407c086-9077-40bf-9663-6f43e75e90cc-istio-token\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:10.576453 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.576288 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2407c086-9077-40bf-9663-6f43e75e90cc-istio-podinfo\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:10.576453 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:10.576296 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2407c086-9077-40bf-9663-6f43e75e90cc-workload-certs\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:11.428307 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.428273 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd"] Jul 13 22:23:11.428860 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.428843 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2407c086-9077-40bf-9663-6f43e75e90cc" containerName="istio-proxy" Jul 13 22:23:11.428910 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.428863 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2407c086-9077-40bf-9663-6f43e75e90cc" containerName="istio-proxy" Jul 13 22:23:11.428998 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.428987 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2407c086-9077-40bf-9663-6f43e75e90cc" containerName="istio-proxy" Jul 13 22:23:11.431225 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.431207 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" Jul 13 22:23:11.434376 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.434313 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3114120a-serving-cert\"" Jul 13 22:23:11.460825 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.460798 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd"] Jul 13 22:23:11.486027 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.486002 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1-maas-api-tls\") pod \"maas-api-e2e-ait-3114120a-cf988fff4-l9kqd\" (UID: \"ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" Jul 13 22:23:11.486162 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.486053 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1-openshift-service-ca\") pod \"maas-api-e2e-ait-3114120a-cf988fff4-l9kqd\" (UID: \"ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" Jul 13 22:23:11.486162 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.486119 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4npmb\" (UniqueName: \"kubernetes.io/projected/ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1-kube-api-access-4npmb\") pod \"maas-api-e2e-ait-3114120a-cf988fff4-l9kqd\" (UID: \"ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" Jul 13 22:23:11.514695 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.514653 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" event={"ID":"7d234f3e-a3e1-42f3-a599-3266de2d7a75","Type":"ContainerStarted","Data":"c3015619ced218b4b9d727c6fca6a855b1d95b49cf41afbead5e1853fd6957a4"} Jul 13 22:23:11.548261 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.548217 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" podStartSLOduration=2.548201841 podStartE2EDuration="2.548201841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:23:11.546767089 +0000 UTC m=+1504.203971858" watchObservedRunningTime="2026-07-13 22:23:11.548201841 +0000 UTC m=+1504.205406609" Jul 13 22:23:11.586885 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.586860 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1-openshift-service-ca\") pod \"maas-api-e2e-ait-3114120a-cf988fff4-l9kqd\" (UID: \"ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" Jul 13 22:23:11.587030 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.586909 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4npmb\" (UniqueName: \"kubernetes.io/projected/ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1-kube-api-access-4npmb\") pod \"maas-api-e2e-ait-3114120a-cf988fff4-l9kqd\" (UID: \"ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" Jul 13 22:23:11.587103 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.587027 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1-maas-api-tls\") pod \"maas-api-e2e-ait-3114120a-cf988fff4-l9kqd\" (UID: \"ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" Jul 13 22:23:11.587647 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.587626 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1-openshift-service-ca\") pod \"maas-api-e2e-ait-3114120a-cf988fff4-l9kqd\" (UID: \"ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" Jul 13 22:23:11.589396 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.589363 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1-maas-api-tls\") pod \"maas-api-e2e-ait-3114120a-cf988fff4-l9kqd\" (UID: \"ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" Jul 13 22:23:11.595723 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.595700 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4npmb\" (UniqueName: \"kubernetes.io/projected/ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1-kube-api-access-4npmb\") pod \"maas-api-e2e-ait-3114120a-cf988fff4-l9kqd\" (UID: \"ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" Jul 13 22:23:11.741642 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.741614 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" Jul 13 22:23:11.931761 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:11.931733 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2407c086-9077-40bf-9663-6f43e75e90cc" path="/var/lib/kubelet/pods/2407c086-9077-40bf-9663-6f43e75e90cc/volumes" Jul 13 22:23:12.085764 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:12.084830 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd"] Jul 13 22:23:12.296157 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:12.296125 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:12.301282 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:12.301253 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:12.520700 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:12.520666 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" event={"ID":"ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1","Type":"ContainerStarted","Data":"eda0270c42f0b7e0d4600f61352762e6aed3e295b637c5e0be12b2b027eaddf8"} Jul 13 22:23:12.521124 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:12.521018 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:12.522196 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:12.522168 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:13.527497 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:13.527459 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" event={"ID":"ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1","Type":"ContainerStarted","Data":"fd431b9c331ccaa131cbf50972ccf2014ddaf68bd42efbe32624039b408ecd59"} Jul 13 22:23:13.527908 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:13.527571 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" Jul 13 22:23:13.549441 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:13.549396 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" podStartSLOduration=2.031683446 podStartE2EDuration="2.549374032s" totalImagesPullingTime="517.690586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:23:13.546740779 +0000 UTC m=+1506.203945546" watchObservedRunningTime="2026-07-13 22:23:13.549374032 +0000 UTC m=+1506.206578800" Jul 13 22:23:16.728767 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:16.728734 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd"] Jul 13 22:23:16.729239 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:16.728957 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" podUID="ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1" containerName="maas-api" containerID="cri-o://fd431b9c331ccaa131cbf50972ccf2014ddaf68bd42efbe32624039b408ecd59" gracePeriod=30 Jul 13 22:23:17.015152 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.015124 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" Jul 13 22:23:17.138229 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.138196 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1-openshift-service-ca\") pod \"ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1\" (UID: \"ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1\") " Jul 13 22:23:17.138398 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.138270 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1-maas-api-tls\") pod \"ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1\" (UID: \"ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1\") " Jul 13 22:23:17.138398 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.138298 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4npmb\" (UniqueName: \"kubernetes.io/projected/ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1-kube-api-access-4npmb\") pod \"ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1\" (UID: \"ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1\") " Jul 13 22:23:17.138638 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.138612 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1" (UID: "ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:23:17.140292 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.140261 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1-kube-api-access-4npmb" (OuterVolumeSpecName: "kube-api-access-4npmb") pod "ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1" (UID: "ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1"). InnerVolumeSpecName "kube-api-access-4npmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:23:17.140376 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.140293 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1" (UID: "ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:23:17.239317 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.239222 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1-openshift-service-ca\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:17.239317 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.239267 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1-maas-api-tls\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:17.239317 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.239276 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4npmb\" (UniqueName: \"kubernetes.io/projected/ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1-kube-api-access-4npmb\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:17.548494 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.548404 2569 generic.go:356] "Generic (PLEG): container finished" podID="ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1" containerID="fd431b9c331ccaa131cbf50972ccf2014ddaf68bd42efbe32624039b408ecd59" exitCode=0 Jul 13 22:23:17.548494 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.548450 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" Jul 13 22:23:17.548714 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.548496 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" event={"ID":"ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1","Type":"ContainerDied","Data":"fd431b9c331ccaa131cbf50972ccf2014ddaf68bd42efbe32624039b408ecd59"} Jul 13 22:23:17.548714 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.548569 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd" event={"ID":"ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1","Type":"ContainerDied","Data":"eda0270c42f0b7e0d4600f61352762e6aed3e295b637c5e0be12b2b027eaddf8"} Jul 13 22:23:17.548714 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.548596 2569 scope.go:117] "RemoveContainer" containerID="fd431b9c331ccaa131cbf50972ccf2014ddaf68bd42efbe32624039b408ecd59" Jul 13 22:23:17.557990 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.557976 2569 scope.go:117] "RemoveContainer" containerID="fd431b9c331ccaa131cbf50972ccf2014ddaf68bd42efbe32624039b408ecd59" Jul 13 22:23:17.558222 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:23:17.558206 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd431b9c331ccaa131cbf50972ccf2014ddaf68bd42efbe32624039b408ecd59\": container with ID starting with fd431b9c331ccaa131cbf50972ccf2014ddaf68bd42efbe32624039b408ecd59 not found: ID does not exist" containerID="fd431b9c331ccaa131cbf50972ccf2014ddaf68bd42efbe32624039b408ecd59" Jul 13 22:23:17.558268 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.558230 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd431b9c331ccaa131cbf50972ccf2014ddaf68bd42efbe32624039b408ecd59"} err="failed to get container status \"fd431b9c331ccaa131cbf50972ccf2014ddaf68bd42efbe32624039b408ecd59\": rpc error: code = NotFound desc = could not find container \"fd431b9c331ccaa131cbf50972ccf2014ddaf68bd42efbe32624039b408ecd59\": container with ID starting with fd431b9c331ccaa131cbf50972ccf2014ddaf68bd42efbe32624039b408ecd59 not found: ID does not exist" Jul 13 22:23:17.571303 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.571275 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd"] Jul 13 22:23:17.581330 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.581293 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3114120a-cf988fff4-l9kqd"] Jul 13 22:23:17.934026 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:17.933929 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1" path="/var/lib/kubelet/pods/ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1/volumes" Jul 13 22:23:22.294888 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:22.294853 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n"] Jul 13 22:23:22.295334 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:22.295085 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" podUID="7d234f3e-a3e1-42f3-a599-3266de2d7a75" containerName="istio-proxy" containerID="cri-o://c3015619ced218b4b9d727c6fca6a855b1d95b49cf41afbead5e1853fd6957a4" gracePeriod=30 Jul 13 22:23:23.455841 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.455812 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:23.499724 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.499693 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-credential-socket\") pod \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " Jul 13 22:23:23.499909 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.499751 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-workload-socket\") pod \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " Jul 13 22:23:23.499909 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.499810 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-data\") pod \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " Jul 13 22:23:23.499909 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.499830 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istiod-ca-cert\") pod \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " Jul 13 22:23:23.499909 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.499900 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-token\") pod \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " Jul 13 22:23:23.500131 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.499922 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwwb2\" (UniqueName: \"kubernetes.io/projected/7d234f3e-a3e1-42f3-a599-3266de2d7a75-kube-api-access-nwwb2\") pod \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " Jul 13 22:23:23.500131 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.499964 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-envoy\") pod \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " Jul 13 22:23:23.500131 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.499998 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-workload-certs\") pod \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " Jul 13 22:23:23.500131 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.500029 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-podinfo\") pod \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\" (UID: \"7d234f3e-a3e1-42f3-a599-3266de2d7a75\") " Jul 13 22:23:23.500131 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.500032 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7d234f3e-a3e1-42f3-a599-3266de2d7a75" (UID: "7d234f3e-a3e1-42f3-a599-3266de2d7a75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:23:23.500402 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.500130 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7d234f3e-a3e1-42f3-a599-3266de2d7a75" (UID: "7d234f3e-a3e1-42f3-a599-3266de2d7a75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:23:23.500468 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.500405 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7d234f3e-a3e1-42f3-a599-3266de2d7a75" (UID: "7d234f3e-a3e1-42f3-a599-3266de2d7a75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:23:23.500468 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.500413 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-credential-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:23.500468 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.500443 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-workload-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:23.500716 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.500689 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7d234f3e-a3e1-42f3-a599-3266de2d7a75" (UID: "7d234f3e-a3e1-42f3-a599-3266de2d7a75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:23:23.501713 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.501673 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-data" (OuterVolumeSpecName: "istio-data") pod "7d234f3e-a3e1-42f3-a599-3266de2d7a75" (UID: "7d234f3e-a3e1-42f3-a599-3266de2d7a75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:23:23.503068 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.502963 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-token" (OuterVolumeSpecName: "istio-token") pod "7d234f3e-a3e1-42f3-a599-3266de2d7a75" (UID: "7d234f3e-a3e1-42f3-a599-3266de2d7a75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:23:23.503327 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.503100 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7d234f3e-a3e1-42f3-a599-3266de2d7a75" (UID: "7d234f3e-a3e1-42f3-a599-3266de2d7a75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:23:23.503327 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.503206 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7d234f3e-a3e1-42f3-a599-3266de2d7a75" (UID: "7d234f3e-a3e1-42f3-a599-3266de2d7a75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:23:23.503890 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.503866 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d234f3e-a3e1-42f3-a599-3266de2d7a75-kube-api-access-nwwb2" (OuterVolumeSpecName: "kube-api-access-nwwb2") pod "7d234f3e-a3e1-42f3-a599-3266de2d7a75" (UID: "7d234f3e-a3e1-42f3-a599-3266de2d7a75"). InnerVolumeSpecName "kube-api-access-nwwb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:23:23.580717 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.580687 2569 generic.go:356] "Generic (PLEG): container finished" podID="7d234f3e-a3e1-42f3-a599-3266de2d7a75" containerID="c3015619ced218b4b9d727c6fca6a855b1d95b49cf41afbead5e1853fd6957a4" exitCode=0 Jul 13 22:23:23.580860 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.580739 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" Jul 13 22:23:23.580860 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.580734 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" event={"ID":"7d234f3e-a3e1-42f3-a599-3266de2d7a75","Type":"ContainerDied","Data":"c3015619ced218b4b9d727c6fca6a855b1d95b49cf41afbead5e1853fd6957a4"} Jul 13 22:23:23.580978 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.580874 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n" event={"ID":"7d234f3e-a3e1-42f3-a599-3266de2d7a75","Type":"ContainerDied","Data":"211ef3f842da5843e57508a6f74555852a0f324b4a89e5152568ada808d07818"} Jul 13 22:23:23.580978 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.580911 2569 scope.go:117] "RemoveContainer" containerID="c3015619ced218b4b9d727c6fca6a855b1d95b49cf41afbead5e1853fd6957a4" Jul 13 22:23:23.591349 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.591333 2569 scope.go:117] "RemoveContainer" containerID="c3015619ced218b4b9d727c6fca6a855b1d95b49cf41afbead5e1853fd6957a4" Jul 13 22:23:23.591618 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:23:23.591599 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3015619ced218b4b9d727c6fca6a855b1d95b49cf41afbead5e1853fd6957a4\": container with ID starting with c3015619ced218b4b9d727c6fca6a855b1d95b49cf41afbead5e1853fd6957a4 not found: ID does not exist" containerID="c3015619ced218b4b9d727c6fca6a855b1d95b49cf41afbead5e1853fd6957a4" Jul 13 22:23:23.591706 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.591624 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3015619ced218b4b9d727c6fca6a855b1d95b49cf41afbead5e1853fd6957a4"} err="failed to get container status \"c3015619ced218b4b9d727c6fca6a855b1d95b49cf41afbead5e1853fd6957a4\": rpc error: code = NotFound desc = could not find container \"c3015619ced218b4b9d727c6fca6a855b1d95b49cf41afbead5e1853fd6957a4\": container with ID starting with c3015619ced218b4b9d727c6fca6a855b1d95b49cf41afbead5e1853fd6957a4 not found: ID does not exist" Jul 13 22:23:23.601673 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.601652 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-data\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:23.601673 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.601673 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istiod-ca-cert\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:23.601798 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.601681 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-token\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:23.601798 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.601690 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwwb2\" (UniqueName: \"kubernetes.io/projected/7d234f3e-a3e1-42f3-a599-3266de2d7a75-kube-api-access-nwwb2\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:23.601798 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.601698 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-envoy\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:23.601798 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.601707 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d234f3e-a3e1-42f3-a599-3266de2d7a75-workload-certs\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:23.601798 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.601714 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d234f3e-a3e1-42f3-a599-3266de2d7a75-istio-podinfo\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:23.605065 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.605042 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n"] Jul 13 22:23:23.610059 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.610038 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3114120a-openshift-default-7f7c89d985-j2v8n"] Jul 13 22:23:23.931150 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:23.931053 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d234f3e-a3e1-42f3-a599-3266de2d7a75" path="/var/lib/kubelet/pods/7d234f3e-a3e1-42f3-a599-3266de2d7a75/volumes" Jul 13 22:23:29.053177 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.053145 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv"] Jul 13 22:23:29.053753 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.053733 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1" containerName="maas-api" Jul 13 22:23:29.053840 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.053755 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1" containerName="maas-api" Jul 13 22:23:29.053840 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.053771 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d234f3e-a3e1-42f3-a599-3266de2d7a75" containerName="istio-proxy" Jul 13 22:23:29.053840 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.053779 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d234f3e-a3e1-42f3-a599-3266de2d7a75" containerName="istio-proxy" Jul 13 22:23:29.053993 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.053882 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce6abda7-716a-4fd1-8eaf-bfb939cdc1c1" containerName="maas-api" Jul 13 22:23:29.053993 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.053896 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d234f3e-a3e1-42f3-a599-3266de2d7a75" containerName="istio-proxy" Jul 13 22:23:29.056830 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.056807 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.060998 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.060974 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-8a36d8be-openshift-default-dockercfg-tjdlm\"" Jul 13 22:23:29.085347 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.085320 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv"] Jul 13 22:23:29.153112 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.153079 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-credential-socket\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.153282 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.153125 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-podinfo\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.153282 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.153156 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkg8q\" (UniqueName: \"kubernetes.io/projected/e53a2bd8-4c7b-4bab-b361-7f490d485f92-kube-api-access-bkg8q\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.153358 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.153272 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istiod-ca-cert\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.153358 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.153306 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-data\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.153358 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.153327 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-envoy\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.153358 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.153350 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-workload-certs\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.153515 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.153378 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-token\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.153515 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.153414 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-workload-socket\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.254660 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.254626 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istiod-ca-cert\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.254660 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.254668 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-data\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.254896 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.254687 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-envoy\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.254896 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.254708 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-workload-certs\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.254896 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.254733 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-token\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.254896 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.254754 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-workload-socket\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.254896 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.254784 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-credential-socket\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.254896 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.254822 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-podinfo\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.255190 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.255103 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkg8q\" (UniqueName: \"kubernetes.io/projected/e53a2bd8-4c7b-4bab-b361-7f490d485f92-kube-api-access-bkg8q\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.255190 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.255109 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-data\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.255190 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.255139 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-credential-socket\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.255342 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.255221 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-workload-certs\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.255342 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.255283 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-workload-socket\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.255513 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.255494 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istiod-ca-cert\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.257111 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.257092 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-envoy\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.257204 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.257186 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-podinfo\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.267610 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.267554 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-token\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.270707 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.270684 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkg8q\" (UniqueName: \"kubernetes.io/projected/e53a2bd8-4c7b-4bab-b361-7f490d485f92-kube-api-access-bkg8q\") pod \"e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.367098 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.367067 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:29.713468 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.713440 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv"] Jul 13 22:23:29.714773 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:23:29.714747 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53a2bd8_4c7b_4bab_b361_7f490d485f92.slice/crio-5d3f5bd1eed5ad597f7c9a16b8f2ef7935bd2175bc92098f5aae0f366487ea29 WatchSource:0}: Error finding container 5d3f5bd1eed5ad597f7c9a16b8f2ef7935bd2175bc92098f5aae0f366487ea29: Status 404 returned error can't find the container with id 5d3f5bd1eed5ad597f7c9a16b8f2ef7935bd2175bc92098f5aae0f366487ea29 Jul 13 22:23:29.716787 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.716749 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:23:29.716866 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.716818 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:23:29.716866 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:29.716844 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:23:30.285256 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:30.285221 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz"] Jul 13 22:23:30.288142 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:30.288122 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" Jul 13 22:23:30.290993 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:30.290971 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-8a36d8be-serving-cert\"" Jul 13 22:23:30.300372 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:30.300308 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz"] Jul 13 22:23:30.367068 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:30.366993 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbnch\" (UniqueName: \"kubernetes.io/projected/2d7779db-ef33-4e15-9fad-fe70b91dcb49-kube-api-access-tbnch\") pod \"maas-api-e2e-derive-8a36d8be-65f967d45-hbffz\" (UID: \"2d7779db-ef33-4e15-9fad-fe70b91dcb49\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" Jul 13 22:23:30.367068 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:30.367025 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7779db-ef33-4e15-9fad-fe70b91dcb49-openshift-service-ca\") pod \"maas-api-e2e-derive-8a36d8be-65f967d45-hbffz\" (UID: \"2d7779db-ef33-4e15-9fad-fe70b91dcb49\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" Jul 13 22:23:30.367068 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:30.367047 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d7779db-ef33-4e15-9fad-fe70b91dcb49-maas-api-tls\") pod \"maas-api-e2e-derive-8a36d8be-65f967d45-hbffz\" (UID: \"2d7779db-ef33-4e15-9fad-fe70b91dcb49\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" Jul 13 22:23:30.468011 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:30.467975 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbnch\" (UniqueName: \"kubernetes.io/projected/2d7779db-ef33-4e15-9fad-fe70b91dcb49-kube-api-access-tbnch\") pod \"maas-api-e2e-derive-8a36d8be-65f967d45-hbffz\" (UID: \"2d7779db-ef33-4e15-9fad-fe70b91dcb49\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" Jul 13 22:23:30.468011 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:30.468012 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7779db-ef33-4e15-9fad-fe70b91dcb49-openshift-service-ca\") pod \"maas-api-e2e-derive-8a36d8be-65f967d45-hbffz\" (UID: \"2d7779db-ef33-4e15-9fad-fe70b91dcb49\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" Jul 13 22:23:30.468221 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:30.468029 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d7779db-ef33-4e15-9fad-fe70b91dcb49-maas-api-tls\") pod \"maas-api-e2e-derive-8a36d8be-65f967d45-hbffz\" (UID: \"2d7779db-ef33-4e15-9fad-fe70b91dcb49\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" Jul 13 22:23:30.468221 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:23:30.468125 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-serving-cert: secret "maas-api-e2e-derive-8a36d8be-serving-cert" not found Jul 13 22:23:30.468221 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:23:30.468191 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d7779db-ef33-4e15-9fad-fe70b91dcb49-maas-api-tls podName:2d7779db-ef33-4e15-9fad-fe70b91dcb49 nodeName:}" failed. No retries permitted until 2026-07-13 22:23:30.968172463 +0000 UTC m=+1523.625377210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d7779db-ef33-4e15-9fad-fe70b91dcb49-maas-api-tls") pod "maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" (UID: "2d7779db-ef33-4e15-9fad-fe70b91dcb49") : secret "maas-api-e2e-derive-8a36d8be-serving-cert" not found Jul 13 22:23:30.468828 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:30.468805 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7779db-ef33-4e15-9fad-fe70b91dcb49-openshift-service-ca\") pod \"maas-api-e2e-derive-8a36d8be-65f967d45-hbffz\" (UID: \"2d7779db-ef33-4e15-9fad-fe70b91dcb49\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" Jul 13 22:23:30.479613 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:30.479588 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbnch\" (UniqueName: \"kubernetes.io/projected/2d7779db-ef33-4e15-9fad-fe70b91dcb49-kube-api-access-tbnch\") pod \"maas-api-e2e-derive-8a36d8be-65f967d45-hbffz\" (UID: \"2d7779db-ef33-4e15-9fad-fe70b91dcb49\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" Jul 13 22:23:30.617362 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:30.617263 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" event={"ID":"e53a2bd8-4c7b-4bab-b361-7f490d485f92","Type":"ContainerStarted","Data":"8462087ed492eede6e517931e746894ef9dfcf0206736d4a144ca354372b9af5"} Jul 13 22:23:30.617362 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:30.617330 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" event={"ID":"e53a2bd8-4c7b-4bab-b361-7f490d485f92","Type":"ContainerStarted","Data":"5d3f5bd1eed5ad597f7c9a16b8f2ef7935bd2175bc92098f5aae0f366487ea29"} Jul 13 22:23:30.652037 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:30.651984 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" podStartSLOduration=1.651970215 podStartE2EDuration="1.651970215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:23:30.649581047 +0000 UTC m=+1523.306785816" watchObservedRunningTime="2026-07-13 22:23:30.651970215 +0000 UTC m=+1523.309174982" Jul 13 22:23:30.973425 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:30.973381 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d7779db-ef33-4e15-9fad-fe70b91dcb49-maas-api-tls\") pod \"maas-api-e2e-derive-8a36d8be-65f967d45-hbffz\" (UID: \"2d7779db-ef33-4e15-9fad-fe70b91dcb49\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" Jul 13 22:23:30.975762 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:30.975738 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d7779db-ef33-4e15-9fad-fe70b91dcb49-maas-api-tls\") pod \"maas-api-e2e-derive-8a36d8be-65f967d45-hbffz\" (UID: \"2d7779db-ef33-4e15-9fad-fe70b91dcb49\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" Jul 13 22:23:31.202248 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:31.202209 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" Jul 13 22:23:31.348947 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:31.348915 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz"] Jul 13 22:23:31.350816 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:23:31.350790 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7779db_ef33_4e15_9fad_fe70b91dcb49.slice/crio-8e9b7b35ff3cdc380c273208d19362aace2915c724afbb4dc58d7e74418c28c0 WatchSource:0}: Error finding container 8e9b7b35ff3cdc380c273208d19362aace2915c724afbb4dc58d7e74418c28c0: Status 404 returned error can't find the container with id 8e9b7b35ff3cdc380c273208d19362aace2915c724afbb4dc58d7e74418c28c0 Jul 13 22:23:31.367969 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:31.367933 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:31.372758 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:31.372730 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:31.624435 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:31.624338 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" event={"ID":"2d7779db-ef33-4e15-9fad-fe70b91dcb49","Type":"ContainerStarted","Data":"8e9b7b35ff3cdc380c273208d19362aace2915c724afbb4dc58d7e74418c28c0"} Jul 13 22:23:31.624628 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:31.624513 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:31.625729 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:31.625706 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:32.631431 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:32.631393 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" event={"ID":"2d7779db-ef33-4e15-9fad-fe70b91dcb49","Type":"ContainerStarted","Data":"fccfe150d9543723eb1901bbeeb9173eb575da4180a8435d1dd5e8c361cd328c"} Jul 13 22:23:32.631857 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:32.631494 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" Jul 13 22:23:32.651587 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:32.651529 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" podStartSLOduration=2.245925481 podStartE2EDuration="2.651514042s" totalImagesPullingTime="405.588561ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:23:32.65005673 +0000 UTC m=+1525.307261498" watchObservedRunningTime="2026-07-13 22:23:32.651514042 +0000 UTC m=+1525.308718810" Jul 13 22:23:34.747206 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:34.747176 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv"] Jul 13 22:23:34.747635 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:34.747370 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" podUID="e53a2bd8-4c7b-4bab-b361-7f490d485f92" containerName="istio-proxy" containerID="cri-o://8462087ed492eede6e517931e746894ef9dfcf0206736d4a144ca354372b9af5" gracePeriod=30 Jul 13 22:23:35.140215 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.140117 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz"] Jul 13 22:23:35.140480 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.140358 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" podUID="2d7779db-ef33-4e15-9fad-fe70b91dcb49" containerName="maas-api" containerID="cri-o://fccfe150d9543723eb1901bbeeb9173eb575da4180a8435d1dd5e8c361cd328c" gracePeriod=30 Jul 13 22:23:35.398357 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.398297 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" Jul 13 22:23:35.519466 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.519433 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7779db-ef33-4e15-9fad-fe70b91dcb49-openshift-service-ca\") pod \"2d7779db-ef33-4e15-9fad-fe70b91dcb49\" (UID: \"2d7779db-ef33-4e15-9fad-fe70b91dcb49\") " Jul 13 22:23:35.519651 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.519500 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbnch\" (UniqueName: \"kubernetes.io/projected/2d7779db-ef33-4e15-9fad-fe70b91dcb49-kube-api-access-tbnch\") pod \"2d7779db-ef33-4e15-9fad-fe70b91dcb49\" (UID: \"2d7779db-ef33-4e15-9fad-fe70b91dcb49\") " Jul 13 22:23:35.519651 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.519570 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d7779db-ef33-4e15-9fad-fe70b91dcb49-maas-api-tls\") pod \"2d7779db-ef33-4e15-9fad-fe70b91dcb49\" (UID: \"2d7779db-ef33-4e15-9fad-fe70b91dcb49\") " Jul 13 22:23:35.519878 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.519851 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7779db-ef33-4e15-9fad-fe70b91dcb49-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2d7779db-ef33-4e15-9fad-fe70b91dcb49" (UID: "2d7779db-ef33-4e15-9fad-fe70b91dcb49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:23:35.521762 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.521737 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d7779db-ef33-4e15-9fad-fe70b91dcb49-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2d7779db-ef33-4e15-9fad-fe70b91dcb49" (UID: "2d7779db-ef33-4e15-9fad-fe70b91dcb49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:23:35.521867 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.521743 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7779db-ef33-4e15-9fad-fe70b91dcb49-kube-api-access-tbnch" (OuterVolumeSpecName: "kube-api-access-tbnch") pod "2d7779db-ef33-4e15-9fad-fe70b91dcb49" (UID: "2d7779db-ef33-4e15-9fad-fe70b91dcb49"). InnerVolumeSpecName "kube-api-access-tbnch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:23:35.620934 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.620906 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7779db-ef33-4e15-9fad-fe70b91dcb49-openshift-service-ca\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:35.620934 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.620931 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbnch\" (UniqueName: \"kubernetes.io/projected/2d7779db-ef33-4e15-9fad-fe70b91dcb49-kube-api-access-tbnch\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:35.621076 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.620941 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d7779db-ef33-4e15-9fad-fe70b91dcb49-maas-api-tls\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:35.647846 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.647819 2569 generic.go:356] "Generic (PLEG): container finished" podID="2d7779db-ef33-4e15-9fad-fe70b91dcb49" containerID="fccfe150d9543723eb1901bbeeb9173eb575da4180a8435d1dd5e8c361cd328c" exitCode=0 Jul 13 22:23:35.647974 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.647861 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" Jul 13 22:23:35.647974 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.647911 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" event={"ID":"2d7779db-ef33-4e15-9fad-fe70b91dcb49","Type":"ContainerDied","Data":"fccfe150d9543723eb1901bbeeb9173eb575da4180a8435d1dd5e8c361cd328c"} Jul 13 22:23:35.647974 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.647966 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz" event={"ID":"2d7779db-ef33-4e15-9fad-fe70b91dcb49","Type":"ContainerDied","Data":"8e9b7b35ff3cdc380c273208d19362aace2915c724afbb4dc58d7e74418c28c0"} Jul 13 22:23:35.648145 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.647998 2569 scope.go:117] "RemoveContainer" containerID="fccfe150d9543723eb1901bbeeb9173eb575da4180a8435d1dd5e8c361cd328c" Jul 13 22:23:35.657071 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.657051 2569 scope.go:117] "RemoveContainer" containerID="fccfe150d9543723eb1901bbeeb9173eb575da4180a8435d1dd5e8c361cd328c" Jul 13 22:23:35.657331 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:23:35.657308 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fccfe150d9543723eb1901bbeeb9173eb575da4180a8435d1dd5e8c361cd328c\": container with ID starting with fccfe150d9543723eb1901bbeeb9173eb575da4180a8435d1dd5e8c361cd328c not found: ID does not exist" containerID="fccfe150d9543723eb1901bbeeb9173eb575da4180a8435d1dd5e8c361cd328c" Jul 13 22:23:35.657394 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.657340 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fccfe150d9543723eb1901bbeeb9173eb575da4180a8435d1dd5e8c361cd328c"} err="failed to get container status \"fccfe150d9543723eb1901bbeeb9173eb575da4180a8435d1dd5e8c361cd328c\": rpc error: code = NotFound desc = could not find container \"fccfe150d9543723eb1901bbeeb9173eb575da4180a8435d1dd5e8c361cd328c\": container with ID starting with fccfe150d9543723eb1901bbeeb9173eb575da4180a8435d1dd5e8c361cd328c not found: ID does not exist" Jul 13 22:23:35.681049 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.681028 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz"] Jul 13 22:23:35.688634 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.688612 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8a36d8be-65f967d45-hbffz"] Jul 13 22:23:35.930666 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:35.930592 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d7779db-ef33-4e15-9fad-fe70b91dcb49" path="/var/lib/kubelet/pods/2d7779db-ef33-4e15-9fad-fe70b91dcb49/volumes" Jul 13 22:23:40.006131 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.006061 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:40.156970 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.156939 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-envoy\") pod \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " Jul 13 22:23:40.156970 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.156975 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkg8q\" (UniqueName: \"kubernetes.io/projected/e53a2bd8-4c7b-4bab-b361-7f490d485f92-kube-api-access-bkg8q\") pod \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " Jul 13 22:23:40.157212 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.157008 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-workload-socket\") pod \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " Jul 13 22:23:40.157212 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.157039 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istiod-ca-cert\") pod \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " Jul 13 22:23:40.157212 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.157156 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-data\") pod \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " Jul 13 22:23:40.157380 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.157224 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-workload-certs\") pod \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " Jul 13 22:23:40.157380 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.157278 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-credential-socket\") pod \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " Jul 13 22:23:40.157380 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.157321 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-podinfo\") pod \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " Jul 13 22:23:40.157380 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.157345 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-token\") pod \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\" (UID: \"e53a2bd8-4c7b-4bab-b361-7f490d485f92\") " Jul 13 22:23:40.157380 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.157370 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e53a2bd8-4c7b-4bab-b361-7f490d485f92" (UID: "e53a2bd8-4c7b-4bab-b361-7f490d485f92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:23:40.157685 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.157479 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-data" (OuterVolumeSpecName: "istio-data") pod "e53a2bd8-4c7b-4bab-b361-7f490d485f92" (UID: "e53a2bd8-4c7b-4bab-b361-7f490d485f92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:23:40.157685 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.157497 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e53a2bd8-4c7b-4bab-b361-7f490d485f92" (UID: "e53a2bd8-4c7b-4bab-b361-7f490d485f92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:23:40.157685 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.157639 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e53a2bd8-4c7b-4bab-b361-7f490d485f92" (UID: "e53a2bd8-4c7b-4bab-b361-7f490d485f92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:23:40.157849 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.157724 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istiod-ca-cert\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:40.157849 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.157745 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-data\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:40.157849 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.157759 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-credential-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:40.157849 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.157773 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-workload-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:40.157849 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.157832 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e53a2bd8-4c7b-4bab-b361-7f490d485f92" (UID: "e53a2bd8-4c7b-4bab-b361-7f490d485f92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:23:40.159374 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.159353 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e53a2bd8-4c7b-4bab-b361-7f490d485f92" (UID: "e53a2bd8-4c7b-4bab-b361-7f490d485f92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:23:40.159475 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.159407 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-token" (OuterVolumeSpecName: "istio-token") pod "e53a2bd8-4c7b-4bab-b361-7f490d485f92" (UID: "e53a2bd8-4c7b-4bab-b361-7f490d485f92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:23:40.159570 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.159523 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53a2bd8-4c7b-4bab-b361-7f490d485f92-kube-api-access-bkg8q" (OuterVolumeSpecName: "kube-api-access-bkg8q") pod "e53a2bd8-4c7b-4bab-b361-7f490d485f92" (UID: "e53a2bd8-4c7b-4bab-b361-7f490d485f92"). InnerVolumeSpecName "kube-api-access-bkg8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:23:40.159639 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.159565 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e53a2bd8-4c7b-4bab-b361-7f490d485f92" (UID: "e53a2bd8-4c7b-4bab-b361-7f490d485f92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:23:40.258366 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.258309 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-podinfo\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:40.258366 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.258332 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-token\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:40.258366 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.258341 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-istio-envoy\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:40.258366 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.258350 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkg8q\" (UniqueName: \"kubernetes.io/projected/e53a2bd8-4c7b-4bab-b361-7f490d485f92-kube-api-access-bkg8q\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:40.258366 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.258359 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e53a2bd8-4c7b-4bab-b361-7f490d485f92-workload-certs\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:23:40.675713 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.675618 2569 generic.go:356] "Generic (PLEG): container finished" podID="e53a2bd8-4c7b-4bab-b361-7f490d485f92" containerID="8462087ed492eede6e517931e746894ef9dfcf0206736d4a144ca354372b9af5" exitCode=0 Jul 13 22:23:40.675713 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.675688 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" Jul 13 22:23:40.675927 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.675711 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" event={"ID":"e53a2bd8-4c7b-4bab-b361-7f490d485f92","Type":"ContainerDied","Data":"8462087ed492eede6e517931e746894ef9dfcf0206736d4a144ca354372b9af5"} Jul 13 22:23:40.675927 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.675787 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv" event={"ID":"e53a2bd8-4c7b-4bab-b361-7f490d485f92","Type":"ContainerDied","Data":"5d3f5bd1eed5ad597f7c9a16b8f2ef7935bd2175bc92098f5aae0f366487ea29"} Jul 13 22:23:40.675927 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.675810 2569 scope.go:117] "RemoveContainer" containerID="8462087ed492eede6e517931e746894ef9dfcf0206736d4a144ca354372b9af5" Jul 13 22:23:40.686297 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.686282 2569 scope.go:117] "RemoveContainer" containerID="8462087ed492eede6e517931e746894ef9dfcf0206736d4a144ca354372b9af5" Jul 13 22:23:40.686613 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:23:40.686567 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8462087ed492eede6e517931e746894ef9dfcf0206736d4a144ca354372b9af5\": container with ID starting with 8462087ed492eede6e517931e746894ef9dfcf0206736d4a144ca354372b9af5 not found: ID does not exist" containerID="8462087ed492eede6e517931e746894ef9dfcf0206736d4a144ca354372b9af5" Jul 13 22:23:40.686711 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.686621 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8462087ed492eede6e517931e746894ef9dfcf0206736d4a144ca354372b9af5"} err="failed to get container status \"8462087ed492eede6e517931e746894ef9dfcf0206736d4a144ca354372b9af5\": rpc error: code = NotFound desc = could not find container \"8462087ed492eede6e517931e746894ef9dfcf0206736d4a144ca354372b9af5\": container with ID starting with 8462087ed492eede6e517931e746894ef9dfcf0206736d4a144ca354372b9af5 not found: ID does not exist" Jul 13 22:23:40.698617 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.698588 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv"] Jul 13 22:23:40.702038 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:40.702018 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-8a36d8be-openshift-default-657d6bc8ff-mdplv"] Jul 13 22:23:41.930998 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:23:41.930963 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e53a2bd8-4c7b-4bab-b361-7f490d485f92" path="/var/lib/kubelet/pods/e53a2bd8-4c7b-4bab-b361-7f490d485f92/volumes" Jul 13 22:25:19.095843 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.095807 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9"] Jul 13 22:25:19.096420 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.096403 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d7779db-ef33-4e15-9fad-fe70b91dcb49" containerName="maas-api" Jul 13 22:25:19.096499 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.096422 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d7779db-ef33-4e15-9fad-fe70b91dcb49" containerName="maas-api" Jul 13 22:25:19.096499 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.096447 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e53a2bd8-4c7b-4bab-b361-7f490d485f92" containerName="istio-proxy" Jul 13 22:25:19.096499 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.096456 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e53a2bd8-4c7b-4bab-b361-7f490d485f92" containerName="istio-proxy" Jul 13 22:25:19.096696 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.096563 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d7779db-ef33-4e15-9fad-fe70b91dcb49" containerName="maas-api" Jul 13 22:25:19.096696 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.096580 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e53a2bd8-4c7b-4bab-b361-7f490d485f92" containerName="istio-proxy" Jul 13 22:25:19.099916 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.099895 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.103429 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.103406 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3cb0ac36-openshift-default-dockercfg-crbw6\"" Jul 13 22:25:19.125185 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.125160 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9"] Jul 13 22:25:19.186318 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.186281 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-token\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.186486 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.186367 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-credential-socket\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.186486 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.186405 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-workload-socket\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.186486 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.186427 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-workload-certs\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.186486 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.186459 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a6a19e-d368-41ee-ae31-97d352d115df-istiod-ca-cert\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.186486 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.186484 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-podinfo\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.186743 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.186576 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tvnz\" (UniqueName: \"kubernetes.io/projected/a1a6a19e-d368-41ee-ae31-97d352d115df-kube-api-access-6tvnz\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.186743 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.186610 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-envoy\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.186743 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.186660 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-data\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.288068 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.288032 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-data\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.288270 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.288084 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-token\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.288270 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.288143 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-credential-socket\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.288270 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.288188 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-workload-socket\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.288270 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.288205 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-workload-certs\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.288270 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.288231 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a6a19e-d368-41ee-ae31-97d352d115df-istiod-ca-cert\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.288563 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.288297 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-podinfo\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.288646 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.288325 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tvnz\" (UniqueName: \"kubernetes.io/projected/a1a6a19e-d368-41ee-ae31-97d352d115df-kube-api-access-6tvnz\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.288706 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.288673 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-envoy\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.288706 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.288402 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-data\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.288821 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.288617 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-workload-certs\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.288821 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.288572 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-workload-socket\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.289061 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.289035 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a6a19e-d368-41ee-ae31-97d352d115df-istiod-ca-cert\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.289237 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.289212 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-credential-socket\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.291288 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.291257 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-envoy\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.291448 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.291427 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-podinfo\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.296724 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.296698 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-token\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.297132 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.297112 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tvnz\" (UniqueName: \"kubernetes.io/projected/a1a6a19e-d368-41ee-ae31-97d352d115df-kube-api-access-6tvnz\") pod \"e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.413508 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.413427 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:19.599208 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.599179 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9"] Jul 13 22:25:19.602056 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:25:19.602027 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a6a19e_d368_41ee_ae31_97d352d115df.slice/crio-306987d21c904ff29c3d12d74c235438c20a1ff56dd1d85810991daf796afe83 WatchSource:0}: Error finding container 306987d21c904ff29c3d12d74c235438c20a1ff56dd1d85810991daf796afe83: Status 404 returned error can't find the container with id 306987d21c904ff29c3d12d74c235438c20a1ff56dd1d85810991daf796afe83 Jul 13 22:25:19.604085 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.604053 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:25:19.604166 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.604123 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:25:19.604166 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:19.604151 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:25:20.165412 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:20.165365 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" event={"ID":"a1a6a19e-d368-41ee-ae31-97d352d115df","Type":"ContainerStarted","Data":"6a10273154c86801aa6db86e1bc0ddccd59068f30b5a195d464414aa9c107578"} Jul 13 22:25:20.165849 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:20.165435 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" event={"ID":"a1a6a19e-d368-41ee-ae31-97d352d115df","Type":"ContainerStarted","Data":"306987d21c904ff29c3d12d74c235438c20a1ff56dd1d85810991daf796afe83"} Jul 13 22:25:20.190836 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:20.190792 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" podStartSLOduration=1.190775492 podStartE2EDuration="1.190775492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:25:20.188588243 +0000 UTC m=+1632.845793011" watchObservedRunningTime="2026-07-13 22:25:20.190775492 +0000 UTC m=+1632.847980260" Jul 13 22:25:20.413843 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:20.413808 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:20.418750 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:20.418690 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:21.171775 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:21.171748 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:21.172735 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:21.172712 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:22.366030 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:22.365994 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm"] Jul 13 22:25:22.369991 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:22.369972 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" Jul 13 22:25:22.372801 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:22.372778 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3cb0ac36-serving-cert\"" Jul 13 22:25:22.383122 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:22.383084 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm"] Jul 13 22:25:22.517886 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:22.517854 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jkkv\" (UniqueName: \"kubernetes.io/projected/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-kube-api-access-9jkkv\") pod \"maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm\" (UID: \"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" Jul 13 22:25:22.518050 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:22.517907 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-maas-api-tls\") pod \"maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm\" (UID: \"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" Jul 13 22:25:22.518050 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:22.517969 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-openshift-service-ca\") pod \"maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm\" (UID: \"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" Jul 13 22:25:22.618892 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:22.618821 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-openshift-service-ca\") pod \"maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm\" (UID: \"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" Jul 13 22:25:22.618892 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:22.618887 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jkkv\" (UniqueName: \"kubernetes.io/projected/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-kube-api-access-9jkkv\") pod \"maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm\" (UID: \"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" Jul 13 22:25:22.619081 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:22.618930 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-maas-api-tls\") pod \"maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm\" (UID: \"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" Jul 13 22:25:22.619081 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:25:22.619025 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-serving-cert: secret "maas-api-e2e-mt-3cb0ac36-serving-cert" not found Jul 13 22:25:22.619081 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:25:22.619076 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-maas-api-tls podName:cd0e22f9-a1a4-4d19-9520-f6e36dc5a534 nodeName:}" failed. No retries permitted until 2026-07-13 22:25:23.119060993 +0000 UTC m=+1635.776265738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-maas-api-tls") pod "maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" (UID: "cd0e22f9-a1a4-4d19-9520-f6e36dc5a534") : secret "maas-api-e2e-mt-3cb0ac36-serving-cert" not found Jul 13 22:25:22.619609 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:22.619585 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-openshift-service-ca\") pod \"maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm\" (UID: \"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" Jul 13 22:25:22.628962 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:22.628934 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jkkv\" (UniqueName: \"kubernetes.io/projected/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-kube-api-access-9jkkv\") pod \"maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm\" (UID: \"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" Jul 13 22:25:23.124230 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:23.124190 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-maas-api-tls\") pod \"maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm\" (UID: \"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" Jul 13 22:25:23.126585 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:23.126562 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-maas-api-tls\") pod \"maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm\" (UID: \"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" Jul 13 22:25:23.281585 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:23.281547 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" Jul 13 22:25:23.422695 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:23.422671 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm"] Jul 13 22:25:23.424373 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:25:23.424347 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0e22f9_a1a4_4d19_9520_f6e36dc5a534.slice/crio-31910da723f10462a630ae7c0578115a8e57780d385402e2c83a1ab4f14877c6 WatchSource:0}: Error finding container 31910da723f10462a630ae7c0578115a8e57780d385402e2c83a1ab4f14877c6: Status 404 returned error can't find the container with id 31910da723f10462a630ae7c0578115a8e57780d385402e2c83a1ab4f14877c6 Jul 13 22:25:24.192704 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:24.192556 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" event={"ID":"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534","Type":"ContainerStarted","Data":"83750b0568b780fed34f2d22bb7a439d893982897910a52ad7d0847effd9b5b5"} Jul 13 22:25:24.192704 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:24.192632 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" event={"ID":"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534","Type":"ContainerStarted","Data":"31910da723f10462a630ae7c0578115a8e57780d385402e2c83a1ab4f14877c6"} Jul 13 22:25:24.192704 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:24.192664 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" Jul 13 22:25:24.210843 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:24.210790 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" podStartSLOduration=1.601046676 podStartE2EDuration="2.210774401s" totalImagesPullingTime="609.727725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:25:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:25:24.210616545 +0000 UTC m=+1636.867821313" watchObservedRunningTime="2026-07-13 22:25:24.210774401 +0000 UTC m=+1636.867979169" Jul 13 22:25:30.209529 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:30.209490 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" Jul 13 22:25:35.018808 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:35.018766 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm"] Jul 13 22:25:35.019305 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:35.019016 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" podUID="cd0e22f9-a1a4-4d19-9520-f6e36dc5a534" containerName="maas-api" containerID="cri-o://83750b0568b780fed34f2d22bb7a439d893982897910a52ad7d0847effd9b5b5" gracePeriod=30 Jul 13 22:25:35.252528 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:35.252491 2569 generic.go:356] "Generic (PLEG): container finished" podID="cd0e22f9-a1a4-4d19-9520-f6e36dc5a534" containerID="83750b0568b780fed34f2d22bb7a439d893982897910a52ad7d0847effd9b5b5" exitCode=0 Jul 13 22:25:35.252528 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:35.252578 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" event={"ID":"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534","Type":"ContainerDied","Data":"83750b0568b780fed34f2d22bb7a439d893982897910a52ad7d0847effd9b5b5"} Jul 13 22:25:35.312217 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:35.312183 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" Jul 13 22:25:35.333644 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:35.333614 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-maas-api-tls\") pod \"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534\" (UID: \"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534\") " Jul 13 22:25:35.333815 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:35.333679 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jkkv\" (UniqueName: \"kubernetes.io/projected/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-kube-api-access-9jkkv\") pod \"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534\" (UID: \"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534\") " Jul 13 22:25:35.333890 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:35.333817 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-openshift-service-ca\") pod \"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534\" (UID: \"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534\") " Jul 13 22:25:35.334464 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:35.334403 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd0e22f9-a1a4-4d19-9520-f6e36dc5a534" (UID: "cd0e22f9-a1a4-4d19-9520-f6e36dc5a534"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:25:35.337097 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:35.337070 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd0e22f9-a1a4-4d19-9520-f6e36dc5a534" (UID: "cd0e22f9-a1a4-4d19-9520-f6e36dc5a534"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:25:35.337240 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:35.337108 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-kube-api-access-9jkkv" (OuterVolumeSpecName: "kube-api-access-9jkkv") pod "cd0e22f9-a1a4-4d19-9520-f6e36dc5a534" (UID: "cd0e22f9-a1a4-4d19-9520-f6e36dc5a534"). InnerVolumeSpecName "kube-api-access-9jkkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:25:35.435056 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:35.435021 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-maas-api-tls\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:25:35.435056 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:35.435050 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jkkv\" (UniqueName: \"kubernetes.io/projected/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-kube-api-access-9jkkv\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:25:35.435056 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:35.435059 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534-openshift-service-ca\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:25:36.259093 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:36.259017 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" Jul 13 22:25:36.259499 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:36.259019 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" event={"ID":"cd0e22f9-a1a4-4d19-9520-f6e36dc5a534","Type":"ContainerDied","Data":"31910da723f10462a630ae7c0578115a8e57780d385402e2c83a1ab4f14877c6"} Jul 13 22:25:36.259499 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:36.259155 2569 scope.go:117] "RemoveContainer" containerID="83750b0568b780fed34f2d22bb7a439d893982897910a52ad7d0847effd9b5b5" Jul 13 22:25:36.277611 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:36.277583 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm"] Jul 13 22:25:36.283245 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:36.283225 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm"] Jul 13 22:25:36.799007 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:36.798903 2569 patch_prober.go:28] interesting pod/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": read tcp 10.132.0.2:38152->10.132.0.58:15021: read: connection reset by peer" start-of-body= Jul 13 22:25:36.799007 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:36.798966 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" podUID="a1a6a19e-d368-41ee-ae31-97d352d115df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": read tcp 10.132.0.2:38152->10.132.0.58:15021: read: connection reset by peer" Jul 13 22:25:37.266680 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:37.266648 2569 generic.go:356] "Generic (PLEG): container finished" podID="a1a6a19e-d368-41ee-ae31-97d352d115df" containerID="6a10273154c86801aa6db86e1bc0ddccd59068f30b5a195d464414aa9c107578" exitCode=0 Jul 13 22:25:37.267042 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:37.266721 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" event={"ID":"a1a6a19e-d368-41ee-ae31-97d352d115df","Type":"ContainerDied","Data":"6a10273154c86801aa6db86e1bc0ddccd59068f30b5a195d464414aa9c107578"} Jul 13 22:25:37.267091 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:37.267069 2569 scope.go:117] "RemoveContainer" containerID="6a10273154c86801aa6db86e1bc0ddccd59068f30b5a195d464414aa9c107578" Jul 13 22:25:37.267990 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:37.267965 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:25:37.268046 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:37.268031 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:25:37.268086 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:37.268061 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:25:37.931495 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:37.931461 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd0e22f9-a1a4-4d19-9520-f6e36dc5a534" path="/var/lib/kubelet/pods/cd0e22f9-a1a4-4d19-9520-f6e36dc5a534/volumes" Jul 13 22:25:38.199466 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:38.199431 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cb0ac36-564cbbbfcc-f9kgm" podUID="cd0e22f9-a1a4-4d19-9520-f6e36dc5a534" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.59:8443/health\": context deadline exceeded" Jul 13 22:25:38.274683 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:38.274649 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" event={"ID":"a1a6a19e-d368-41ee-ae31-97d352d115df","Type":"ContainerStarted","Data":"2dea6a6f64c25cef342fdda06fa6a187c9c2dbfc288824d831c10c7745dfb14f"} Jul 13 22:25:38.413721 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:38.413677 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:38.413979 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:38.413957 2569 patch_prober.go:28] interesting pod/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 13 22:25:38.414025 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:38.414004 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" podUID="a1a6a19e-d368-41ee-ae31-97d352d115df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 13 22:25:39.414169 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:39.414135 2569 patch_prober.go:28] interesting pod/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 13 22:25:39.414583 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:39.414200 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" podUID="a1a6a19e-d368-41ee-ae31-97d352d115df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 13 22:25:40.414241 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:40.414206 2569 patch_prober.go:28] interesting pod/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 13 22:25:40.414676 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:40.414268 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" podUID="a1a6a19e-d368-41ee-ae31-97d352d115df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 13 22:25:41.259058 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:41.259026 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9"] Jul 13 22:25:41.259242 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:41.259219 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" podUID="a1a6a19e-d368-41ee-ae31-97d352d115df" containerName="istio-proxy" containerID="cri-o://2dea6a6f64c25cef342fdda06fa6a187c9c2dbfc288824d831c10c7745dfb14f" gracePeriod=30 Jul 13 22:25:47.720752 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.720728 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:47.857278 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.857208 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-workload-certs\") pod \"a1a6a19e-d368-41ee-ae31-97d352d115df\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " Jul 13 22:25:47.857278 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.857261 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-podinfo\") pod \"a1a6a19e-d368-41ee-ae31-97d352d115df\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " Jul 13 22:25:47.857496 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.857286 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-workload-socket\") pod \"a1a6a19e-d368-41ee-ae31-97d352d115df\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " Jul 13 22:25:47.857496 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.857306 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-data\") pod \"a1a6a19e-d368-41ee-ae31-97d352d115df\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " Jul 13 22:25:47.857496 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.857331 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-credential-socket\") pod \"a1a6a19e-d368-41ee-ae31-97d352d115df\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " Jul 13 22:25:47.857496 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.857350 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tvnz\" (UniqueName: \"kubernetes.io/projected/a1a6a19e-d368-41ee-ae31-97d352d115df-kube-api-access-6tvnz\") pod \"a1a6a19e-d368-41ee-ae31-97d352d115df\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " Jul 13 22:25:47.857496 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.857377 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a6a19e-d368-41ee-ae31-97d352d115df-istiod-ca-cert\") pod \"a1a6a19e-d368-41ee-ae31-97d352d115df\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " Jul 13 22:25:47.857496 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.857452 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-token\") pod \"a1a6a19e-d368-41ee-ae31-97d352d115df\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " Jul 13 22:25:47.857496 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.857479 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-envoy\") pod \"a1a6a19e-d368-41ee-ae31-97d352d115df\" (UID: \"a1a6a19e-d368-41ee-ae31-97d352d115df\") " Jul 13 22:25:47.857910 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.857491 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a1a6a19e-d368-41ee-ae31-97d352d115df" (UID: "a1a6a19e-d368-41ee-ae31-97d352d115df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:25:47.857910 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.857665 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a1a6a19e-d368-41ee-ae31-97d352d115df" (UID: "a1a6a19e-d368-41ee-ae31-97d352d115df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:25:47.857910 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.857719 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a1a6a19e-d368-41ee-ae31-97d352d115df" (UID: "a1a6a19e-d368-41ee-ae31-97d352d115df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:25:47.857910 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.857785 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-workload-certs\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:25:47.857910 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.857806 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-workload-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:25:47.858193 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.857929 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a6a19e-d368-41ee-ae31-97d352d115df-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a1a6a19e-d368-41ee-ae31-97d352d115df" (UID: "a1a6a19e-d368-41ee-ae31-97d352d115df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:25:47.858405 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.858368 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-data" (OuterVolumeSpecName: "istio-data") pod "a1a6a19e-d368-41ee-ae31-97d352d115df" (UID: "a1a6a19e-d368-41ee-ae31-97d352d115df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:25:47.859613 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.859587 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a1a6a19e-d368-41ee-ae31-97d352d115df" (UID: "a1a6a19e-d368-41ee-ae31-97d352d115df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:25:47.859721 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.859637 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-token" (OuterVolumeSpecName: "istio-token") pod "a1a6a19e-d368-41ee-ae31-97d352d115df" (UID: "a1a6a19e-d368-41ee-ae31-97d352d115df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:25:47.859780 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.859718 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a1a6a19e-d368-41ee-ae31-97d352d115df" (UID: "a1a6a19e-d368-41ee-ae31-97d352d115df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:25:47.859780 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.859726 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a6a19e-d368-41ee-ae31-97d352d115df-kube-api-access-6tvnz" (OuterVolumeSpecName: "kube-api-access-6tvnz") pod "a1a6a19e-d368-41ee-ae31-97d352d115df" (UID: "a1a6a19e-d368-41ee-ae31-97d352d115df"). InnerVolumeSpecName "kube-api-access-6tvnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:25:47.958866 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.958839 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-token\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:25:47.958866 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.958862 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-envoy\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:25:47.958993 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.958870 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-podinfo\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:25:47.958993 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.958878 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-istio-data\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:25:47.958993 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.958887 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a6a19e-d368-41ee-ae31-97d352d115df-credential-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:25:47.958993 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.958898 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tvnz\" (UniqueName: \"kubernetes.io/projected/a1a6a19e-d368-41ee-ae31-97d352d115df-kube-api-access-6tvnz\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:25:47.958993 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:47.958912 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a6a19e-d368-41ee-ae31-97d352d115df-istiod-ca-cert\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:25:48.344174 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:48.344139 2569 generic.go:356] "Generic (PLEG): container finished" podID="a1a6a19e-d368-41ee-ae31-97d352d115df" containerID="2dea6a6f64c25cef342fdda06fa6a187c9c2dbfc288824d831c10c7745dfb14f" exitCode=0 Jul 13 22:25:48.344338 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:48.344189 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" Jul 13 22:25:48.344338 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:48.344228 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" event={"ID":"a1a6a19e-d368-41ee-ae31-97d352d115df","Type":"ContainerDied","Data":"2dea6a6f64c25cef342fdda06fa6a187c9c2dbfc288824d831c10c7745dfb14f"} Jul 13 22:25:48.344338 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:48.344287 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9" event={"ID":"a1a6a19e-d368-41ee-ae31-97d352d115df","Type":"ContainerDied","Data":"306987d21c904ff29c3d12d74c235438c20a1ff56dd1d85810991daf796afe83"} Jul 13 22:25:48.344338 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:48.344312 2569 scope.go:117] "RemoveContainer" containerID="2dea6a6f64c25cef342fdda06fa6a187c9c2dbfc288824d831c10c7745dfb14f" Jul 13 22:25:48.353059 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:48.353022 2569 scope.go:117] "RemoveContainer" containerID="6a10273154c86801aa6db86e1bc0ddccd59068f30b5a195d464414aa9c107578" Jul 13 22:25:48.360551 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:48.360523 2569 scope.go:117] "RemoveContainer" containerID="2dea6a6f64c25cef342fdda06fa6a187c9c2dbfc288824d831c10c7745dfb14f" Jul 13 22:25:48.360807 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:25:48.360787 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dea6a6f64c25cef342fdda06fa6a187c9c2dbfc288824d831c10c7745dfb14f\": container with ID starting with 2dea6a6f64c25cef342fdda06fa6a187c9c2dbfc288824d831c10c7745dfb14f not found: ID does not exist" containerID="2dea6a6f64c25cef342fdda06fa6a187c9c2dbfc288824d831c10c7745dfb14f" Jul 13 22:25:48.360880 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:48.360818 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dea6a6f64c25cef342fdda06fa6a187c9c2dbfc288824d831c10c7745dfb14f"} err="failed to get container status \"2dea6a6f64c25cef342fdda06fa6a187c9c2dbfc288824d831c10c7745dfb14f\": rpc error: code = NotFound desc = could not find container \"2dea6a6f64c25cef342fdda06fa6a187c9c2dbfc288824d831c10c7745dfb14f\": container with ID starting with 2dea6a6f64c25cef342fdda06fa6a187c9c2dbfc288824d831c10c7745dfb14f not found: ID does not exist" Jul 13 22:25:48.360880 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:48.360845 2569 scope.go:117] "RemoveContainer" containerID="6a10273154c86801aa6db86e1bc0ddccd59068f30b5a195d464414aa9c107578" Jul 13 22:25:48.361056 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:25:48.361040 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a10273154c86801aa6db86e1bc0ddccd59068f30b5a195d464414aa9c107578\": container with ID starting with 6a10273154c86801aa6db86e1bc0ddccd59068f30b5a195d464414aa9c107578 not found: ID does not exist" containerID="6a10273154c86801aa6db86e1bc0ddccd59068f30b5a195d464414aa9c107578" Jul 13 22:25:48.361095 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:48.361063 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a10273154c86801aa6db86e1bc0ddccd59068f30b5a195d464414aa9c107578"} err="failed to get container status \"6a10273154c86801aa6db86e1bc0ddccd59068f30b5a195d464414aa9c107578\": rpc error: code = NotFound desc = could not find container \"6a10273154c86801aa6db86e1bc0ddccd59068f30b5a195d464414aa9c107578\": container with ID starting with 6a10273154c86801aa6db86e1bc0ddccd59068f30b5a195d464414aa9c107578 not found: ID does not exist" Jul 13 22:25:48.364482 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:48.364458 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9"] Jul 13 22:25:48.367310 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:48.367290 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3cb0ac36-openshift-default-594df6fdc-4brm9"] Jul 13 22:25:49.931687 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:25:49.931647 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1a6a19e-d368-41ee-ae31-97d352d115df" path="/var/lib/kubelet/pods/a1a6a19e-d368-41ee-ae31-97d352d115df/volumes" Jul 13 22:26:01.291408 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.291368 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz"] Jul 13 22:26:01.292074 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.292052 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a6a19e-d368-41ee-ae31-97d352d115df" containerName="istio-proxy" Jul 13 22:26:01.292074 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.292073 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a6a19e-d368-41ee-ae31-97d352d115df" containerName="istio-proxy" Jul 13 22:26:01.292235 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.292084 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a6a19e-d368-41ee-ae31-97d352d115df" containerName="istio-proxy" Jul 13 22:26:01.292235 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.292090 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a6a19e-d368-41ee-ae31-97d352d115df" containerName="istio-proxy" Jul 13 22:26:01.292235 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.292110 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd0e22f9-a1a4-4d19-9520-f6e36dc5a534" containerName="maas-api" Jul 13 22:26:01.292235 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.292115 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd0e22f9-a1a4-4d19-9520-f6e36dc5a534" containerName="maas-api" Jul 13 22:26:01.292235 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.292182 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a6a19e-d368-41ee-ae31-97d352d115df" containerName="istio-proxy" Jul 13 22:26:01.292235 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.292192 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a6a19e-d368-41ee-ae31-97d352d115df" containerName="istio-proxy" Jul 13 22:26:01.292235 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.292199 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd0e22f9-a1a4-4d19-9520-f6e36dc5a534" containerName="maas-api" Jul 13 22:26:01.296766 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.296742 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.299904 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.299882 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6caf7710-openshift-default-dockercfg-vbhfx\"" Jul 13 22:26:01.313821 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.313725 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz"] Jul 13 22:26:01.376835 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.376797 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-envoy\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.376999 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.376851 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-workload-socket\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.376999 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.376912 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-workload-certs\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.376999 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.376956 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-data\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.377112 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.377010 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-podinfo\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.377112 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.377029 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-credential-socket\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.377112 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.377050 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5765\" (UniqueName: \"kubernetes.io/projected/dee8cd22-d328-4dad-ba43-5b412107b6e5-kube-api-access-t5765\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.377112 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.377107 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-token\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.377249 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.377205 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee8cd22-d328-4dad-ba43-5b412107b6e5-istiod-ca-cert\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.478132 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.478079 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-token\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.478340 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.478180 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee8cd22-d328-4dad-ba43-5b412107b6e5-istiod-ca-cert\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.478340 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.478210 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-envoy\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.478340 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.478247 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-workload-socket\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.478340 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.478276 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-workload-certs\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.478340 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.478302 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-data\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.478661 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.478361 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-podinfo\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.478661 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.478391 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-credential-socket\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.478661 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.478424 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5765\" (UniqueName: \"kubernetes.io/projected/dee8cd22-d328-4dad-ba43-5b412107b6e5-kube-api-access-t5765\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.479015 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.478987 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee8cd22-d328-4dad-ba43-5b412107b6e5-istiod-ca-cert\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.479375 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.479346 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-data\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.479644 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.479623 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-workload-socket\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.479905 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.479880 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-workload-certs\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.480209 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.480187 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-credential-socket\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.481650 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.481441 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-envoy\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.482296 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.482276 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-podinfo\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.487057 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.487031 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5765\" (UniqueName: \"kubernetes.io/projected/dee8cd22-d328-4dad-ba43-5b412107b6e5-kube-api-access-t5765\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.487273 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.487256 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-token\") pod \"e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.614983 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.614900 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:01.756016 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.755989 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz"] Jul 13 22:26:01.757904 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:26:01.757873 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee8cd22_d328_4dad_ba43_5b412107b6e5.slice/crio-d7933cc6e599b0fd672ba6f50a6b5068041270b56d0257cc414809ed6e22dfaf WatchSource:0}: Error finding container d7933cc6e599b0fd672ba6f50a6b5068041270b56d0257cc414809ed6e22dfaf: Status 404 returned error can't find the container with id d7933cc6e599b0fd672ba6f50a6b5068041270b56d0257cc414809ed6e22dfaf Jul 13 22:26:01.760044 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.760013 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:26:01.760132 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.760074 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:26:01.760132 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:01.760101 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:26:02.419650 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:02.419597 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" event={"ID":"dee8cd22-d328-4dad-ba43-5b412107b6e5","Type":"ContainerStarted","Data":"0f7f7573ae7648cc7a922ea2fe27afcd081c10946c4a9d90a5207182db0d4f36"} Jul 13 22:26:02.420054 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:02.419672 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" event={"ID":"dee8cd22-d328-4dad-ba43-5b412107b6e5","Type":"ContainerStarted","Data":"d7933cc6e599b0fd672ba6f50a6b5068041270b56d0257cc414809ed6e22dfaf"} Jul 13 22:26:02.446239 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:02.446193 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" podStartSLOduration=1.44617665 podStartE2EDuration="1.44617665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:26:02.440417441 +0000 UTC m=+1675.097622209" watchObservedRunningTime="2026-07-13 22:26:02.44617665 +0000 UTC m=+1675.103381417" Jul 13 22:26:02.616069 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:02.616036 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:02.620849 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:02.620827 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:03.425660 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:03.425631 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:03.426723 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:03.426702 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:04.556329 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:04.556292 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm"] Jul 13 22:26:04.560595 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:04.560574 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" Jul 13 22:26:04.563047 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:04.563024 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6caf7710-serving-cert\"" Jul 13 22:26:04.573939 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:04.573916 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm"] Jul 13 22:26:04.708914 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:04.708874 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvtb2\" (UniqueName: \"kubernetes.io/projected/ca27ee96-398b-472f-a12a-37a882ba386c-kube-api-access-zvtb2\") pod \"maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm\" (UID: \"ca27ee96-398b-472f-a12a-37a882ba386c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" Jul 13 22:26:04.709081 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:04.708938 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca27ee96-398b-472f-a12a-37a882ba386c-maas-api-tls\") pod \"maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm\" (UID: \"ca27ee96-398b-472f-a12a-37a882ba386c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" Jul 13 22:26:04.709081 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:04.708964 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca27ee96-398b-472f-a12a-37a882ba386c-openshift-service-ca\") pod \"maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm\" (UID: \"ca27ee96-398b-472f-a12a-37a882ba386c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" Jul 13 22:26:04.809894 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:04.809796 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvtb2\" (UniqueName: \"kubernetes.io/projected/ca27ee96-398b-472f-a12a-37a882ba386c-kube-api-access-zvtb2\") pod \"maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm\" (UID: \"ca27ee96-398b-472f-a12a-37a882ba386c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" Jul 13 22:26:04.809894 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:04.809847 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca27ee96-398b-472f-a12a-37a882ba386c-maas-api-tls\") pod \"maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm\" (UID: \"ca27ee96-398b-472f-a12a-37a882ba386c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" Jul 13 22:26:04.809894 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:04.809865 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca27ee96-398b-472f-a12a-37a882ba386c-openshift-service-ca\") pod \"maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm\" (UID: \"ca27ee96-398b-472f-a12a-37a882ba386c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" Jul 13 22:26:04.810178 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:26:04.809947 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-serving-cert: secret "maas-api-e2e-mt-6caf7710-serving-cert" not found Jul 13 22:26:04.810178 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:26:04.810028 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca27ee96-398b-472f-a12a-37a882ba386c-maas-api-tls podName:ca27ee96-398b-472f-a12a-37a882ba386c nodeName:}" failed. No retries permitted until 2026-07-13 22:26:05.31000949 +0000 UTC m=+1677.967214244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ca27ee96-398b-472f-a12a-37a882ba386c-maas-api-tls") pod "maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" (UID: "ca27ee96-398b-472f-a12a-37a882ba386c") : secret "maas-api-e2e-mt-6caf7710-serving-cert" not found Jul 13 22:26:04.810575 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:04.810558 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca27ee96-398b-472f-a12a-37a882ba386c-openshift-service-ca\") pod \"maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm\" (UID: \"ca27ee96-398b-472f-a12a-37a882ba386c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" Jul 13 22:26:04.820213 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:04.820193 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvtb2\" (UniqueName: \"kubernetes.io/projected/ca27ee96-398b-472f-a12a-37a882ba386c-kube-api-access-zvtb2\") pod \"maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm\" (UID: \"ca27ee96-398b-472f-a12a-37a882ba386c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" Jul 13 22:26:05.315994 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:05.315958 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca27ee96-398b-472f-a12a-37a882ba386c-maas-api-tls\") pod \"maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm\" (UID: \"ca27ee96-398b-472f-a12a-37a882ba386c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" Jul 13 22:26:05.318583 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:05.318557 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca27ee96-398b-472f-a12a-37a882ba386c-maas-api-tls\") pod \"maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm\" (UID: \"ca27ee96-398b-472f-a12a-37a882ba386c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" Jul 13 22:26:05.473381 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:05.473352 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" Jul 13 22:26:05.814057 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:05.814029 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm"] Jul 13 22:26:05.816761 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:26:05.816726 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca27ee96_398b_472f_a12a_37a882ba386c.slice/crio-fdd8fc40097bc8e6ab2b2369c04343eb79ad87af01b6bb825f2015d1f9b8725f WatchSource:0}: Error finding container fdd8fc40097bc8e6ab2b2369c04343eb79ad87af01b6bb825f2015d1f9b8725f: Status 404 returned error can't find the container with id fdd8fc40097bc8e6ab2b2369c04343eb79ad87af01b6bb825f2015d1f9b8725f Jul 13 22:26:06.442504 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:06.442462 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" event={"ID":"ca27ee96-398b-472f-a12a-37a882ba386c","Type":"ContainerStarted","Data":"d6b5e7d4b2a2c49a8a1599edb0ef06c23047ad32d30bac510f798d3a59916019"} Jul 13 22:26:06.442730 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:06.442572 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" event={"ID":"ca27ee96-398b-472f-a12a-37a882ba386c","Type":"ContainerStarted","Data":"fdd8fc40097bc8e6ab2b2369c04343eb79ad87af01b6bb825f2015d1f9b8725f"} Jul 13 22:26:06.442730 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:06.442603 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" Jul 13 22:26:06.462279 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:06.462195 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" podStartSLOduration=2.06975758 podStartE2EDuration="2.462182213s" totalImagesPullingTime="392.424633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:26:06.459164773 +0000 UTC m=+1679.116369540" watchObservedRunningTime="2026-07-13 22:26:06.462182213 +0000 UTC m=+1679.119386981" Jul 13 22:26:12.453118 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:12.453090 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" Jul 13 22:26:16.496576 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:16.496512 2569 generic.go:356] "Generic (PLEG): container finished" podID="dee8cd22-d328-4dad-ba43-5b412107b6e5" containerID="0f7f7573ae7648cc7a922ea2fe27afcd081c10946c4a9d90a5207182db0d4f36" exitCode=0 Jul 13 22:26:16.496991 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:16.496590 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" event={"ID":"dee8cd22-d328-4dad-ba43-5b412107b6e5","Type":"ContainerDied","Data":"0f7f7573ae7648cc7a922ea2fe27afcd081c10946c4a9d90a5207182db0d4f36"} Jul 13 22:26:16.496991 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:16.496953 2569 scope.go:117] "RemoveContainer" containerID="0f7f7573ae7648cc7a922ea2fe27afcd081c10946c4a9d90a5207182db0d4f36" Jul 13 22:26:16.498043 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:16.498014 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:26:16.498163 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:16.498081 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:26:16.498163 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:16.498114 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:26:17.506711 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:17.506665 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" event={"ID":"dee8cd22-d328-4dad-ba43-5b412107b6e5","Type":"ContainerStarted","Data":"0dec8fea0b84fa0eb1c2d3be3372032b4215ed2d77a9e96c2e7cbe2742e98264"} Jul 13 22:26:17.615829 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:17.615790 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:17.616804 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:17.616780 2569 patch_prober.go:28] interesting pod/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 13 22:26:17.616861 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:17.616832 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" podUID="dee8cd22-d328-4dad-ba43-5b412107b6e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 13 22:26:18.426395 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:18.426363 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:18.615558 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:18.615496 2569 patch_prober.go:28] interesting pod/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 13 22:26:18.615938 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:18.615572 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" podUID="dee8cd22-d328-4dad-ba43-5b412107b6e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 13 22:26:19.615947 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:19.615910 2569 patch_prober.go:28] interesting pod/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 13 22:26:19.616332 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:19.615991 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" podUID="dee8cd22-d328-4dad-ba43-5b412107b6e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 13 22:26:20.615982 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:20.615946 2569 patch_prober.go:28] interesting pod/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 13 22:26:20.616377 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:20.616001 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" podUID="dee8cd22-d328-4dad-ba43-5b412107b6e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 13 22:26:21.618995 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:21.618960 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:21.619982 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:21.619961 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:26:31.562580 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:31.559269 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm"] Jul 13 22:26:31.564139 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:31.559517 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" podUID="ca27ee96-398b-472f-a12a-37a882ba386c" containerName="maas-api" containerID="cri-o://d6b5e7d4b2a2c49a8a1599edb0ef06c23047ad32d30bac510f798d3a59916019" gracePeriod=30 Jul 13 22:26:31.828792 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:31.828765 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" Jul 13 22:26:31.861955 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:31.861922 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca27ee96-398b-472f-a12a-37a882ba386c-openshift-service-ca\") pod \"ca27ee96-398b-472f-a12a-37a882ba386c\" (UID: \"ca27ee96-398b-472f-a12a-37a882ba386c\") " Jul 13 22:26:31.862101 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:31.861976 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca27ee96-398b-472f-a12a-37a882ba386c-maas-api-tls\") pod \"ca27ee96-398b-472f-a12a-37a882ba386c\" (UID: \"ca27ee96-398b-472f-a12a-37a882ba386c\") " Jul 13 22:26:31.862101 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:31.862029 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvtb2\" (UniqueName: \"kubernetes.io/projected/ca27ee96-398b-472f-a12a-37a882ba386c-kube-api-access-zvtb2\") pod \"ca27ee96-398b-472f-a12a-37a882ba386c\" (UID: \"ca27ee96-398b-472f-a12a-37a882ba386c\") " Jul 13 22:26:31.862242 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:31.862218 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca27ee96-398b-472f-a12a-37a882ba386c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca27ee96-398b-472f-a12a-37a882ba386c" (UID: "ca27ee96-398b-472f-a12a-37a882ba386c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:26:31.862392 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:31.862377 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca27ee96-398b-472f-a12a-37a882ba386c-openshift-service-ca\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:26:31.864017 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:31.863989 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca27ee96-398b-472f-a12a-37a882ba386c-kube-api-access-zvtb2" (OuterVolumeSpecName: "kube-api-access-zvtb2") pod "ca27ee96-398b-472f-a12a-37a882ba386c" (UID: "ca27ee96-398b-472f-a12a-37a882ba386c"). InnerVolumeSpecName "kube-api-access-zvtb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:26:31.864101 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:31.864039 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca27ee96-398b-472f-a12a-37a882ba386c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca27ee96-398b-472f-a12a-37a882ba386c" (UID: "ca27ee96-398b-472f-a12a-37a882ba386c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:26:31.962898 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:31.962858 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca27ee96-398b-472f-a12a-37a882ba386c-maas-api-tls\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:26:31.962898 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:31.962893 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvtb2\" (UniqueName: \"kubernetes.io/projected/ca27ee96-398b-472f-a12a-37a882ba386c-kube-api-access-zvtb2\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:26:32.594264 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:32.594230 2569 generic.go:356] "Generic (PLEG): container finished" podID="ca27ee96-398b-472f-a12a-37a882ba386c" containerID="d6b5e7d4b2a2c49a8a1599edb0ef06c23047ad32d30bac510f798d3a59916019" exitCode=0 Jul 13 22:26:32.594756 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:32.594275 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" Jul 13 22:26:32.594756 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:32.594311 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" event={"ID":"ca27ee96-398b-472f-a12a-37a882ba386c","Type":"ContainerDied","Data":"d6b5e7d4b2a2c49a8a1599edb0ef06c23047ad32d30bac510f798d3a59916019"} Jul 13 22:26:32.594756 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:32.594365 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm" event={"ID":"ca27ee96-398b-472f-a12a-37a882ba386c","Type":"ContainerDied","Data":"fdd8fc40097bc8e6ab2b2369c04343eb79ad87af01b6bb825f2015d1f9b8725f"} Jul 13 22:26:32.594756 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:32.594391 2569 scope.go:117] "RemoveContainer" containerID="d6b5e7d4b2a2c49a8a1599edb0ef06c23047ad32d30bac510f798d3a59916019" Jul 13 22:26:32.603255 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:32.603238 2569 scope.go:117] "RemoveContainer" containerID="d6b5e7d4b2a2c49a8a1599edb0ef06c23047ad32d30bac510f798d3a59916019" Jul 13 22:26:32.603549 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:26:32.603514 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b5e7d4b2a2c49a8a1599edb0ef06c23047ad32d30bac510f798d3a59916019\": container with ID starting with d6b5e7d4b2a2c49a8a1599edb0ef06c23047ad32d30bac510f798d3a59916019 not found: ID does not exist" containerID="d6b5e7d4b2a2c49a8a1599edb0ef06c23047ad32d30bac510f798d3a59916019" Jul 13 22:26:32.603638 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:32.603575 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b5e7d4b2a2c49a8a1599edb0ef06c23047ad32d30bac510f798d3a59916019"} err="failed to get container status \"d6b5e7d4b2a2c49a8a1599edb0ef06c23047ad32d30bac510f798d3a59916019\": rpc error: code = NotFound desc = could not find container \"d6b5e7d4b2a2c49a8a1599edb0ef06c23047ad32d30bac510f798d3a59916019\": container with ID starting with d6b5e7d4b2a2c49a8a1599edb0ef06c23047ad32d30bac510f798d3a59916019 not found: ID does not exist" Jul 13 22:26:32.614503 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:32.614483 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm"] Jul 13 22:26:32.621809 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:32.621788 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6caf7710-67d5fbd795-zbnsm"] Jul 13 22:26:33.931061 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:26:33.931028 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca27ee96-398b-472f-a12a-37a882ba386c" path="/var/lib/kubelet/pods/ca27ee96-398b-472f-a12a-37a882ba386c/volumes" Jul 13 22:27:26.000133 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:26.000100 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz"] Jul 13 22:27:26.000621 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:26.000332 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" podUID="dee8cd22-d328-4dad-ba43-5b412107b6e5" containerName="istio-proxy" containerID="cri-o://0dec8fea0b84fa0eb1c2d3be3372032b4215ed2d77a9e96c2e7cbe2742e98264" gracePeriod=30 Jul 13 22:27:31.254001 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.253972 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:27:31.374068 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.373968 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-podinfo\") pod \"dee8cd22-d328-4dad-ba43-5b412107b6e5\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " Jul 13 22:27:31.374068 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.374028 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee8cd22-d328-4dad-ba43-5b412107b6e5-istiod-ca-cert\") pod \"dee8cd22-d328-4dad-ba43-5b412107b6e5\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " Jul 13 22:27:31.374068 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.374058 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-envoy\") pod \"dee8cd22-d328-4dad-ba43-5b412107b6e5\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " Jul 13 22:27:31.374361 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.374129 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-credential-socket\") pod \"dee8cd22-d328-4dad-ba43-5b412107b6e5\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " Jul 13 22:27:31.374361 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.374161 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-workload-socket\") pod \"dee8cd22-d328-4dad-ba43-5b412107b6e5\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " Jul 13 22:27:31.374361 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.374245 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5765\" (UniqueName: \"kubernetes.io/projected/dee8cd22-d328-4dad-ba43-5b412107b6e5-kube-api-access-t5765\") pod \"dee8cd22-d328-4dad-ba43-5b412107b6e5\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " Jul 13 22:27:31.374361 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.374280 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-data\") pod \"dee8cd22-d328-4dad-ba43-5b412107b6e5\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " Jul 13 22:27:31.374361 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.374323 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-workload-certs\") pod \"dee8cd22-d328-4dad-ba43-5b412107b6e5\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " Jul 13 22:27:31.374361 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.374348 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-token\") pod \"dee8cd22-d328-4dad-ba43-5b412107b6e5\" (UID: \"dee8cd22-d328-4dad-ba43-5b412107b6e5\") " Jul 13 22:27:31.374730 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.374434 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dee8cd22-d328-4dad-ba43-5b412107b6e5" (UID: "dee8cd22-d328-4dad-ba43-5b412107b6e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:27:31.374730 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.374565 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dee8cd22-d328-4dad-ba43-5b412107b6e5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dee8cd22-d328-4dad-ba43-5b412107b6e5" (UID: "dee8cd22-d328-4dad-ba43-5b412107b6e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:27:31.374730 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.374706 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dee8cd22-d328-4dad-ba43-5b412107b6e5" (UID: "dee8cd22-d328-4dad-ba43-5b412107b6e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:27:31.374892 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.374773 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dee8cd22-d328-4dad-ba43-5b412107b6e5" (UID: "dee8cd22-d328-4dad-ba43-5b412107b6e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:27:31.374892 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.374795 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-credential-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:27:31.374892 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.374814 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-workload-certs\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:27:31.374892 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.374832 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee8cd22-d328-4dad-ba43-5b412107b6e5-istiod-ca-cert\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:27:31.375583 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.375553 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-data" (OuterVolumeSpecName: "istio-data") pod "dee8cd22-d328-4dad-ba43-5b412107b6e5" (UID: "dee8cd22-d328-4dad-ba43-5b412107b6e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:27:31.376459 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.376428 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dee8cd22-d328-4dad-ba43-5b412107b6e5" (UID: "dee8cd22-d328-4dad-ba43-5b412107b6e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:27:31.376588 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.376499 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dee8cd22-d328-4dad-ba43-5b412107b6e5" (UID: "dee8cd22-d328-4dad-ba43-5b412107b6e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:27:31.376588 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.376577 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-token" (OuterVolumeSpecName: "istio-token") pod "dee8cd22-d328-4dad-ba43-5b412107b6e5" (UID: "dee8cd22-d328-4dad-ba43-5b412107b6e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:27:31.376717 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.376701 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee8cd22-d328-4dad-ba43-5b412107b6e5-kube-api-access-t5765" (OuterVolumeSpecName: "kube-api-access-t5765") pod "dee8cd22-d328-4dad-ba43-5b412107b6e5" (UID: "dee8cd22-d328-4dad-ba43-5b412107b6e5"). InnerVolumeSpecName "kube-api-access-t5765". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:27:31.475560 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.475498 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5765\" (UniqueName: \"kubernetes.io/projected/dee8cd22-d328-4dad-ba43-5b412107b6e5-kube-api-access-t5765\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:27:31.475560 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.475530 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-data\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:27:31.475560 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.475565 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-token\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:27:31.475850 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.475572 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-podinfo\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:27:31.475850 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.475581 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-istio-envoy\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:27:31.475850 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.475589 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee8cd22-d328-4dad-ba43-5b412107b6e5-workload-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:27:31.898791 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.898760 2569 generic.go:356] "Generic (PLEG): container finished" podID="dee8cd22-d328-4dad-ba43-5b412107b6e5" containerID="0dec8fea0b84fa0eb1c2d3be3372032b4215ed2d77a9e96c2e7cbe2742e98264" exitCode=0 Jul 13 22:27:31.898981 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.898804 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" event={"ID":"dee8cd22-d328-4dad-ba43-5b412107b6e5","Type":"ContainerDied","Data":"0dec8fea0b84fa0eb1c2d3be3372032b4215ed2d77a9e96c2e7cbe2742e98264"} Jul 13 22:27:31.898981 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.898810 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" Jul 13 22:27:31.898981 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.898878 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz" event={"ID":"dee8cd22-d328-4dad-ba43-5b412107b6e5","Type":"ContainerDied","Data":"d7933cc6e599b0fd672ba6f50a6b5068041270b56d0257cc414809ed6e22dfaf"} Jul 13 22:27:31.898981 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.898927 2569 scope.go:117] "RemoveContainer" containerID="0dec8fea0b84fa0eb1c2d3be3372032b4215ed2d77a9e96c2e7cbe2742e98264" Jul 13 22:27:31.908988 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.908970 2569 scope.go:117] "RemoveContainer" containerID="0f7f7573ae7648cc7a922ea2fe27afcd081c10946c4a9d90a5207182db0d4f36" Jul 13 22:27:31.917336 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.917313 2569 scope.go:117] "RemoveContainer" containerID="0dec8fea0b84fa0eb1c2d3be3372032b4215ed2d77a9e96c2e7cbe2742e98264" Jul 13 22:27:31.917616 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:27:31.917596 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dec8fea0b84fa0eb1c2d3be3372032b4215ed2d77a9e96c2e7cbe2742e98264\": container with ID starting with 0dec8fea0b84fa0eb1c2d3be3372032b4215ed2d77a9e96c2e7cbe2742e98264 not found: ID does not exist" containerID="0dec8fea0b84fa0eb1c2d3be3372032b4215ed2d77a9e96c2e7cbe2742e98264" Jul 13 22:27:31.917658 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.917629 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dec8fea0b84fa0eb1c2d3be3372032b4215ed2d77a9e96c2e7cbe2742e98264"} err="failed to get container status \"0dec8fea0b84fa0eb1c2d3be3372032b4215ed2d77a9e96c2e7cbe2742e98264\": rpc error: code = NotFound desc = could not find container \"0dec8fea0b84fa0eb1c2d3be3372032b4215ed2d77a9e96c2e7cbe2742e98264\": container with ID starting with 0dec8fea0b84fa0eb1c2d3be3372032b4215ed2d77a9e96c2e7cbe2742e98264 not found: ID does not exist" Jul 13 22:27:31.917705 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.917656 2569 scope.go:117] "RemoveContainer" containerID="0f7f7573ae7648cc7a922ea2fe27afcd081c10946c4a9d90a5207182db0d4f36" Jul 13 22:27:31.917919 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:27:31.917898 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f7f7573ae7648cc7a922ea2fe27afcd081c10946c4a9d90a5207182db0d4f36\": container with ID starting with 0f7f7573ae7648cc7a922ea2fe27afcd081c10946c4a9d90a5207182db0d4f36 not found: ID does not exist" containerID="0f7f7573ae7648cc7a922ea2fe27afcd081c10946c4a9d90a5207182db0d4f36" Jul 13 22:27:31.917961 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.917926 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f7f7573ae7648cc7a922ea2fe27afcd081c10946c4a9d90a5207182db0d4f36"} err="failed to get container status \"0f7f7573ae7648cc7a922ea2fe27afcd081c10946c4a9d90a5207182db0d4f36\": rpc error: code = NotFound desc = could not find container \"0f7f7573ae7648cc7a922ea2fe27afcd081c10946c4a9d90a5207182db0d4f36\": container with ID starting with 0f7f7573ae7648cc7a922ea2fe27afcd081c10946c4a9d90a5207182db0d4f36 not found: ID does not exist" Jul 13 22:27:31.923613 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.923590 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz"] Jul 13 22:27:31.931715 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:31.931692 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6caf7710-openshift-default-6d65dbcb8c-c2kkz"] Jul 13 22:27:33.932072 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:27:33.932037 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dee8cd22-d328-4dad-ba43-5b412107b6e5" path="/var/lib/kubelet/pods/dee8cd22-d328-4dad-ba43-5b412107b6e5/volumes" Jul 13 22:28:07.953714 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:07.953682 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 22:28:07.956843 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:07.956823 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 22:28:07.962756 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:07.962738 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 22:28:07.965558 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:07.965523 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 22:28:21.363501 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.363466 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g"] Jul 13 22:28:21.363946 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.363853 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee8cd22-d328-4dad-ba43-5b412107b6e5" containerName="istio-proxy" Jul 13 22:28:21.363946 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.363865 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee8cd22-d328-4dad-ba43-5b412107b6e5" containerName="istio-proxy" Jul 13 22:28:21.363946 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.363891 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca27ee96-398b-472f-a12a-37a882ba386c" containerName="maas-api" Jul 13 22:28:21.363946 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.363896 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca27ee96-398b-472f-a12a-37a882ba386c" containerName="maas-api" Jul 13 22:28:21.364098 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.363970 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dee8cd22-d328-4dad-ba43-5b412107b6e5" containerName="istio-proxy" Jul 13 22:28:21.364098 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.363978 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dee8cd22-d328-4dad-ba43-5b412107b6e5" containerName="istio-proxy" Jul 13 22:28:21.364098 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.363985 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca27ee96-398b-472f-a12a-37a882ba386c" containerName="maas-api" Jul 13 22:28:21.367101 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.367084 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.370301 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.370279 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fd15db-openshift-default-dockercfg-gznc4\"" Jul 13 22:28:21.391794 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.391770 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g"] Jul 13 22:28:21.501179 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.501149 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istiod-ca-cert\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.501339 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.501191 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-workload-certs\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.501339 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.501209 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-envoy\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.501339 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.501240 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-token\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.501339 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.501304 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-workload-socket\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.501491 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.501354 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-data\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.501491 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.501376 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bslkq\" (UniqueName: \"kubernetes.io/projected/92929369-5eeb-46d4-9ba7-5c51b56c9f33-kube-api-access-bslkq\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.501491 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.501420 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-podinfo\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.501491 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.501441 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-credential-socket\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.602136 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.602099 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-podinfo\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.602136 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.602142 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-credential-socket\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.602369 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.602180 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istiod-ca-cert\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.602369 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.602212 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-workload-certs\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.602369 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.602228 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-envoy\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.602369 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.602286 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-token\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.602369 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.602345 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-workload-socket\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.602687 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.602396 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-data\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.602687 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.602423 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bslkq\" (UniqueName: \"kubernetes.io/projected/92929369-5eeb-46d4-9ba7-5c51b56c9f33-kube-api-access-bslkq\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.602820 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.602796 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-workload-certs\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.602881 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.602840 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-workload-socket\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.603018 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.602996 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-data\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.603088 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.603043 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-credential-socket\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.603088 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.603073 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istiod-ca-cert\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.604514 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.604487 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-envoy\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.604665 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.604636 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-podinfo\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.615270 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.615213 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-token\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.615371 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.615342 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bslkq\" (UniqueName: \"kubernetes.io/projected/92929369-5eeb-46d4-9ba7-5c51b56c9f33-kube-api-access-bslkq\") pod \"e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.677239 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.677217 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:21.835048 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.835020 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g"] Jul 13 22:28:21.836065 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:28:21.836039 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92929369_5eeb_46d4_9ba7_5c51b56c9f33.slice/crio-6aa4c9003fd162dfefaf44b52cd2fb0f060b778e03cfd16fe0e99eed3e53439f WatchSource:0}: Error finding container 6aa4c9003fd162dfefaf44b52cd2fb0f060b778e03cfd16fe0e99eed3e53439f: Status 404 returned error can't find the container with id 6aa4c9003fd162dfefaf44b52cd2fb0f060b778e03cfd16fe0e99eed3e53439f Jul 13 22:28:21.838278 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.838237 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:28:21.838356 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.838328 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:28:21.838398 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:21.838369 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:28:22.141821 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:22.141724 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" event={"ID":"92929369-5eeb-46d4-9ba7-5c51b56c9f33","Type":"ContainerStarted","Data":"259ab7fe616399913daf162707e0c0bdf7a220d56a07342b927c228eb90a4500"} Jul 13 22:28:22.141821 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:22.141789 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" event={"ID":"92929369-5eeb-46d4-9ba7-5c51b56c9f33","Type":"ContainerStarted","Data":"6aa4c9003fd162dfefaf44b52cd2fb0f060b778e03cfd16fe0e99eed3e53439f"} Jul 13 22:28:22.173419 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:22.173363 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" podStartSLOduration=1.173348554 podStartE2EDuration="1.173348554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:28:22.173220173 +0000 UTC m=+1814.830424942" watchObservedRunningTime="2026-07-13 22:28:22.173348554 +0000 UTC m=+1814.830553321" Jul 13 22:28:22.678387 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:22.678349 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:22.683837 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:22.683810 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:23.147030 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:23.146995 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:23.147830 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:23.147809 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:24.493642 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:24.493609 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w"] Jul 13 22:28:24.494207 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:24.494192 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee8cd22-d328-4dad-ba43-5b412107b6e5" containerName="istio-proxy" Jul 13 22:28:24.494254 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:24.494210 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee8cd22-d328-4dad-ba43-5b412107b6e5" containerName="istio-proxy" Jul 13 22:28:24.497914 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:24.497896 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" Jul 13 22:28:24.501693 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:24.501524 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-fd15db-serving-cert\"" Jul 13 22:28:24.515891 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:24.515871 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w"] Jul 13 22:28:24.632908 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:24.632874 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e3d6513-5ae2-4684-be2a-33824317157c-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w\" (UID: \"8e3d6513-5ae2-4684-be2a-33824317157c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" Jul 13 22:28:24.633097 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:24.632935 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mxh2\" (UniqueName: \"kubernetes.io/projected/8e3d6513-5ae2-4684-be2a-33824317157c-kube-api-access-4mxh2\") pod \"maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w\" (UID: \"8e3d6513-5ae2-4684-be2a-33824317157c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" Jul 13 22:28:24.633097 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:24.633016 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e3d6513-5ae2-4684-be2a-33824317157c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w\" (UID: \"8e3d6513-5ae2-4684-be2a-33824317157c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" Jul 13 22:28:24.734072 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:24.734035 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mxh2\" (UniqueName: \"kubernetes.io/projected/8e3d6513-5ae2-4684-be2a-33824317157c-kube-api-access-4mxh2\") pod \"maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w\" (UID: \"8e3d6513-5ae2-4684-be2a-33824317157c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" Jul 13 22:28:24.734206 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:24.734084 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e3d6513-5ae2-4684-be2a-33824317157c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w\" (UID: \"8e3d6513-5ae2-4684-be2a-33824317157c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" Jul 13 22:28:24.734206 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:24.734146 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e3d6513-5ae2-4684-be2a-33824317157c-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w\" (UID: \"8e3d6513-5ae2-4684-be2a-33824317157c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" Jul 13 22:28:24.734286 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:28:24.734239 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-serving-cert: secret "maas-api-e2e-inf-a-fd15db-serving-cert" not found Jul 13 22:28:24.734321 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:28:24.734289 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e3d6513-5ae2-4684-be2a-33824317157c-maas-api-tls podName:8e3d6513-5ae2-4684-be2a-33824317157c nodeName:}" failed. No retries permitted until 2026-07-13 22:28:25.234274159 +0000 UTC m=+1817.891478905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8e3d6513-5ae2-4684-be2a-33824317157c-maas-api-tls") pod "maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" (UID: "8e3d6513-5ae2-4684-be2a-33824317157c") : secret "maas-api-e2e-inf-a-fd15db-serving-cert" not found Jul 13 22:28:24.734762 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:24.734743 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e3d6513-5ae2-4684-be2a-33824317157c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w\" (UID: \"8e3d6513-5ae2-4684-be2a-33824317157c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" Jul 13 22:28:24.755923 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:24.755866 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mxh2\" (UniqueName: \"kubernetes.io/projected/8e3d6513-5ae2-4684-be2a-33824317157c-kube-api-access-4mxh2\") pod \"maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w\" (UID: \"8e3d6513-5ae2-4684-be2a-33824317157c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" Jul 13 22:28:25.236949 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:25.236911 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e3d6513-5ae2-4684-be2a-33824317157c-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w\" (UID: \"8e3d6513-5ae2-4684-be2a-33824317157c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" Jul 13 22:28:25.239293 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:25.239268 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e3d6513-5ae2-4684-be2a-33824317157c-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w\" (UID: \"8e3d6513-5ae2-4684-be2a-33824317157c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" Jul 13 22:28:25.408943 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:25.408906 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" Jul 13 22:28:25.565902 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:25.565877 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w"] Jul 13 22:28:25.567590 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:28:25.567557 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e3d6513_5ae2_4684_be2a_33824317157c.slice/crio-ad9dc027646dedd48097534990aeb06ddadd92d514c3a331f17b4723049e5a64 WatchSource:0}: Error finding container ad9dc027646dedd48097534990aeb06ddadd92d514c3a331f17b4723049e5a64: Status 404 returned error can't find the container with id ad9dc027646dedd48097534990aeb06ddadd92d514c3a331f17b4723049e5a64 Jul 13 22:28:25.569025 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:25.569004 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 22:28:26.162857 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:26.162817 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" event={"ID":"8e3d6513-5ae2-4684-be2a-33824317157c","Type":"ContainerStarted","Data":"4cf3dd81c3baa9c8695449f299fbf32468fae20e85713e7ca15c001e8cbb22fd"} Jul 13 22:28:26.163026 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:26.162878 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" event={"ID":"8e3d6513-5ae2-4684-be2a-33824317157c","Type":"ContainerStarted","Data":"ad9dc027646dedd48097534990aeb06ddadd92d514c3a331f17b4723049e5a64"} Jul 13 22:28:26.163026 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:26.162994 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" Jul 13 22:28:29.308673 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.308617 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" podStartSLOduration=4.880596767 podStartE2EDuration="5.308600654s" totalImagesPullingTime="428.003887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:28:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:28:26.189643735 +0000 UTC m=+1818.846848504" watchObservedRunningTime="2026-07-13 22:28:29.308600654 +0000 UTC m=+1821.965805422" Jul 13 22:28:29.309330 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.309308 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx"] Jul 13 22:28:29.313057 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.313036 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.316614 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.316593 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-61b7b8-openshift-default-dockercfg-xwlgq\"" Jul 13 22:28:29.330891 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.330863 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx"] Jul 13 22:28:29.474308 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.474274 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-credential-socket\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.474308 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.474309 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-workload-certs\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.474502 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.474330 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405aa82e-1358-4ffc-8653-eccb3956c435-istio-podinfo\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.474502 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.474404 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-workload-socket\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.474586 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.474520 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-istio-data\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.474586 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.474574 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405aa82e-1358-4ffc-8653-eccb3956c435-istio-token\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.474651 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.474636 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405aa82e-1358-4ffc-8653-eccb3956c435-istiod-ca-cert\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.474685 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.474663 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-istio-envoy\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.474720 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.474695 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7clzv\" (UniqueName: \"kubernetes.io/projected/405aa82e-1358-4ffc-8653-eccb3956c435-kube-api-access-7clzv\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.575627 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.575528 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-credential-socket\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.575627 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.575583 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-workload-certs\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.575830 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.575701 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405aa82e-1358-4ffc-8653-eccb3956c435-istio-podinfo\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.575830 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.575748 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-workload-socket\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.575830 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.575818 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-istio-data\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.575990 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.575845 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405aa82e-1358-4ffc-8653-eccb3956c435-istio-token\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.575990 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.575907 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-workload-certs\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.575990 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.575913 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405aa82e-1358-4ffc-8653-eccb3956c435-istiod-ca-cert\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.575990 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.575940 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-istio-envoy\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.575990 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.575973 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7clzv\" (UniqueName: \"kubernetes.io/projected/405aa82e-1358-4ffc-8653-eccb3956c435-kube-api-access-7clzv\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.576231 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.576138 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-istio-data\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.576231 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.576179 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-workload-socket\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.576416 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.576394 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-credential-socket\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.576530 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.576513 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405aa82e-1358-4ffc-8653-eccb3956c435-istiod-ca-cert\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.578650 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.578627 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405aa82e-1358-4ffc-8653-eccb3956c435-istio-podinfo\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.578851 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.578833 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-istio-envoy\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.588022 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.587995 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405aa82e-1358-4ffc-8653-eccb3956c435-istio-token\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.588309 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.588287 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7clzv\" (UniqueName: \"kubernetes.io/projected/405aa82e-1358-4ffc-8653-eccb3956c435-kube-api-access-7clzv\") pod \"e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.623675 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.623656 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:29.776530 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.776504 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx"] Jul 13 22:28:29.778442 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:28:29.778408 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405aa82e_1358_4ffc_8653_eccb3956c435.slice/crio-aec46df46f6189e4b61b1b3e542622a6a3365c20614a0f216b2a17150a2c9664 WatchSource:0}: Error finding container aec46df46f6189e4b61b1b3e542622a6a3365c20614a0f216b2a17150a2c9664: Status 404 returned error can't find the container with id aec46df46f6189e4b61b1b3e542622a6a3365c20614a0f216b2a17150a2c9664 Jul 13 22:28:29.780984 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.780943 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:28:29.781070 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.781030 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:28:29.781111 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:29.781070 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:28:30.187163 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:30.187123 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" event={"ID":"405aa82e-1358-4ffc-8653-eccb3956c435","Type":"ContainerStarted","Data":"db70aeda20cb3350c6eba806173ea5e02704c4d2eac37f85462a8705cf73f030"} Jul 13 22:28:30.187360 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:30.187193 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" event={"ID":"405aa82e-1358-4ffc-8653-eccb3956c435","Type":"ContainerStarted","Data":"aec46df46f6189e4b61b1b3e542622a6a3365c20614a0f216b2a17150a2c9664"} Jul 13 22:28:30.209687 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:30.209645 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" podStartSLOduration=1.209630916 podStartE2EDuration="1.209630916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:28:30.208609798 +0000 UTC m=+1822.865814566" watchObservedRunningTime="2026-07-13 22:28:30.209630916 +0000 UTC m=+1822.866835757" Jul 13 22:28:30.624019 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:30.623948 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:30.628606 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:30.628579 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:31.191924 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:31.191892 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:31.192817 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:31.192794 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:32.172550 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:32.172508 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" Jul 13 22:28:32.277706 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:32.277659 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc"] Jul 13 22:28:32.281975 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:32.281948 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" Jul 13 22:28:32.285189 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:32.285149 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-61b7b8-serving-cert\"" Jul 13 22:28:32.296885 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:32.296862 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc"] Jul 13 22:28:32.402635 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:32.402589 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kclxq\" (UniqueName: \"kubernetes.io/projected/8b7fc6cf-9e57-493b-987a-963ee5681ae5-kube-api-access-kclxq\") pod \"maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc\" (UID: \"8b7fc6cf-9e57-493b-987a-963ee5681ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" Jul 13 22:28:32.402827 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:32.402691 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7fc6cf-9e57-493b-987a-963ee5681ae5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc\" (UID: \"8b7fc6cf-9e57-493b-987a-963ee5681ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" Jul 13 22:28:32.402827 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:32.402749 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b7fc6cf-9e57-493b-987a-963ee5681ae5-maas-api-tls\") pod \"maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc\" (UID: \"8b7fc6cf-9e57-493b-987a-963ee5681ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" Jul 13 22:28:32.503964 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:32.503932 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kclxq\" (UniqueName: \"kubernetes.io/projected/8b7fc6cf-9e57-493b-987a-963ee5681ae5-kube-api-access-kclxq\") pod \"maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc\" (UID: \"8b7fc6cf-9e57-493b-987a-963ee5681ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" Jul 13 22:28:32.504134 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:32.503984 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7fc6cf-9e57-493b-987a-963ee5681ae5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc\" (UID: \"8b7fc6cf-9e57-493b-987a-963ee5681ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" Jul 13 22:28:32.504178 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:32.504158 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b7fc6cf-9e57-493b-987a-963ee5681ae5-maas-api-tls\") pod \"maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc\" (UID: \"8b7fc6cf-9e57-493b-987a-963ee5681ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" Jul 13 22:28:32.504293 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:28:32.504272 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-serving-cert: secret "maas-api-e2e-inf-b-61b7b8-serving-cert" not found Jul 13 22:28:32.504362 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:28:32.504352 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b7fc6cf-9e57-493b-987a-963ee5681ae5-maas-api-tls podName:8b7fc6cf-9e57-493b-987a-963ee5681ae5 nodeName:}" failed. No retries permitted until 2026-07-13 22:28:33.004329031 +0000 UTC m=+1825.661533778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8b7fc6cf-9e57-493b-987a-963ee5681ae5-maas-api-tls") pod "maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" (UID: "8b7fc6cf-9e57-493b-987a-963ee5681ae5") : secret "maas-api-e2e-inf-b-61b7b8-serving-cert" not found Jul 13 22:28:32.504694 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:32.504677 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7fc6cf-9e57-493b-987a-963ee5681ae5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc\" (UID: \"8b7fc6cf-9e57-493b-987a-963ee5681ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" Jul 13 22:28:32.524147 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:32.524122 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kclxq\" (UniqueName: \"kubernetes.io/projected/8b7fc6cf-9e57-493b-987a-963ee5681ae5-kube-api-access-kclxq\") pod \"maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc\" (UID: \"8b7fc6cf-9e57-493b-987a-963ee5681ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" Jul 13 22:28:33.010308 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:33.010270 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b7fc6cf-9e57-493b-987a-963ee5681ae5-maas-api-tls\") pod \"maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc\" (UID: \"8b7fc6cf-9e57-493b-987a-963ee5681ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" Jul 13 22:28:33.012733 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:33.012712 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b7fc6cf-9e57-493b-987a-963ee5681ae5-maas-api-tls\") pod \"maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc\" (UID: \"8b7fc6cf-9e57-493b-987a-963ee5681ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" Jul 13 22:28:33.196808 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:33.196772 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" Jul 13 22:28:33.337400 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:33.337375 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc"] Jul 13 22:28:33.339431 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:28:33.339403 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7fc6cf_9e57_493b_987a_963ee5681ae5.slice/crio-c76e404392b66f3aa9a0a15216402a40febf26f2466f41faaff2e78d34fc63b7 WatchSource:0}: Error finding container c76e404392b66f3aa9a0a15216402a40febf26f2466f41faaff2e78d34fc63b7: Status 404 returned error can't find the container with id c76e404392b66f3aa9a0a15216402a40febf26f2466f41faaff2e78d34fc63b7 Jul 13 22:28:34.210862 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:34.210823 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" event={"ID":"8b7fc6cf-9e57-493b-987a-963ee5681ae5","Type":"ContainerStarted","Data":"7528848b05901cb945cb22db7428637b54a22be01a0b26e08d69d991867c2f10"} Jul 13 22:28:34.211254 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:34.210873 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" event={"ID":"8b7fc6cf-9e57-493b-987a-963ee5681ae5","Type":"ContainerStarted","Data":"c76e404392b66f3aa9a0a15216402a40febf26f2466f41faaff2e78d34fc63b7"} Jul 13 22:28:34.211254 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:34.210918 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" Jul 13 22:28:34.230606 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:34.230556 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" podStartSLOduration=1.800861751 podStartE2EDuration="2.230521177s" totalImagesPullingTime="429.659426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:28:34.229181765 +0000 UTC m=+1826.886386533" watchObservedRunningTime="2026-07-13 22:28:34.230521177 +0000 UTC m=+1826.887725947" Jul 13 22:28:39.188766 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:39.188715 2569 patch_prober.go:28] interesting pod/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 13 22:28:39.189163 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:39.188797 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" podUID="92929369-5eeb-46d4-9ba7-5c51b56c9f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 13 22:28:40.221732 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:40.221700 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" Jul 13 22:28:41.250486 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:41.250453 2569 generic.go:356] "Generic (PLEG): container finished" podID="92929369-5eeb-46d4-9ba7-5c51b56c9f33" containerID="259ab7fe616399913daf162707e0c0bdf7a220d56a07342b927c228eb90a4500" exitCode=0 Jul 13 22:28:41.250929 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:41.250495 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" event={"ID":"92929369-5eeb-46d4-9ba7-5c51b56c9f33","Type":"ContainerDied","Data":"259ab7fe616399913daf162707e0c0bdf7a220d56a07342b927c228eb90a4500"} Jul 13 22:28:41.250929 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:41.250914 2569 scope.go:117] "RemoveContainer" containerID="259ab7fe616399913daf162707e0c0bdf7a220d56a07342b927c228eb90a4500" Jul 13 22:28:41.252033 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:41.252006 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:28:41.252110 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:41.252074 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:28:41.252110 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:41.252105 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:28:42.268773 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:42.268733 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" event={"ID":"92929369-5eeb-46d4-9ba7-5c51b56c9f33","Type":"ContainerStarted","Data":"3f1f3e501096ca838696af637507efd8c421ed732fab3d6ca56a19f8091fcb42"} Jul 13 22:28:42.678313 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:42.678204 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:42.678510 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:42.678366 2569 patch_prober.go:28] interesting pod/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 13 22:28:42.678510 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:42.678433 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" podUID="92929369-5eeb-46d4-9ba7-5c51b56c9f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 13 22:28:43.678675 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:43.678637 2569 patch_prober.go:28] interesting pod/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 13 22:28:43.679277 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:43.679242 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" podUID="92929369-5eeb-46d4-9ba7-5c51b56c9f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 13 22:28:44.678735 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:44.678692 2569 patch_prober.go:28] interesting pod/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 13 22:28:44.679241 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:44.678773 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" podUID="92929369-5eeb-46d4-9ba7-5c51b56c9f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 13 22:28:45.678172 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:45.678133 2569 patch_prober.go:28] interesting pod/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 13 22:28:45.678462 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:45.678431 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" podUID="92929369-5eeb-46d4-9ba7-5c51b56c9f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 13 22:28:46.682031 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:46.681946 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:46.682481 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:46.682394 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:46.683274 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:46.683250 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:28:47.233783 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:47.233739 2569 patch_prober.go:28] interesting pod/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 13 22:28:47.234108 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:47.233836 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" podUID="405aa82e-1358-4ffc-8653-eccb3956c435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 13 22:28:49.313206 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:49.313178 2569 generic.go:356] "Generic (PLEG): container finished" podID="405aa82e-1358-4ffc-8653-eccb3956c435" containerID="db70aeda20cb3350c6eba806173ea5e02704c4d2eac37f85462a8705cf73f030" exitCode=0 Jul 13 22:28:49.313637 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:49.313258 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" event={"ID":"405aa82e-1358-4ffc-8653-eccb3956c435","Type":"ContainerDied","Data":"db70aeda20cb3350c6eba806173ea5e02704c4d2eac37f85462a8705cf73f030"} Jul 13 22:28:49.313706 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:49.313691 2569 scope.go:117] "RemoveContainer" containerID="db70aeda20cb3350c6eba806173ea5e02704c4d2eac37f85462a8705cf73f030" Jul 13 22:28:49.314721 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:49.314696 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:28:49.314812 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:49.314797 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:28:49.314858 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:49.314845 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:28:50.320221 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:50.320188 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" event={"ID":"405aa82e-1358-4ffc-8653-eccb3956c435","Type":"ContainerStarted","Data":"541fd371e256da5fa2a04b47cdf00d6f69a0984b028e5dd6dcc1433ecc407cd7"} Jul 13 22:28:50.624660 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:50.624573 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:50.624876 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:50.624854 2569 patch_prober.go:28] interesting pod/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 13 22:28:50.624927 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:50.624900 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" podUID="405aa82e-1358-4ffc-8653-eccb3956c435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 13 22:28:51.624097 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:51.624060 2569 patch_prober.go:28] interesting pod/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 13 22:28:51.624526 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:51.624111 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" podUID="405aa82e-1358-4ffc-8653-eccb3956c435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 13 22:28:52.624828 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:52.624792 2569 patch_prober.go:28] interesting pod/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 13 22:28:52.625228 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:52.624847 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" podUID="405aa82e-1358-4ffc-8653-eccb3956c435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 13 22:28:53.624600 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:53.624554 2569 patch_prober.go:28] interesting pod/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 13 22:28:53.624802 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:53.624623 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" podUID="405aa82e-1358-4ffc-8653-eccb3956c435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 13 22:28:54.627980 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:54.627948 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:54.628398 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:54.628212 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:28:54.629002 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:28:54.628980 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:30:00.143558 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.143445 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd15db-29733030-q2xws"] Jul 13 22:30:00.152274 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.152236 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733030-jql45"] Jul 13 22:30:00.153098 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.152596 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd15db-29733030-q2xws" Jul 13 22:30:00.155999 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.155952 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxd86\"" Jul 13 22:30:00.156141 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.156024 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61b7b8-29733030-57z4c"] Jul 13 22:30:00.156390 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.156366 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733030-jql45" Jul 13 22:30:00.158859 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.158801 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61b7b8-29733030-57z4c" Jul 13 22:30:00.159027 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.159007 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd15db-29733030-q2xws"] Jul 13 22:30:00.161831 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.161804 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733030-jql45"] Jul 13 22:30:00.168363 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.168340 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd15db-29733030-q2xws" Jul 13 22:30:00.177477 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.177459 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733030-jql45" Jul 13 22:30:00.186195 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.186135 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61b7b8-29733030-57z4c"] Jul 13 22:30:00.216830 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.216804 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61b7b8-29733030-57z4c" Jul 13 22:30:00.387082 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.387048 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd15db-29733030-q2xws"] Jul 13 22:30:00.390034 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:30:00.390008 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc81fe52_6827_4fbd_a2bd_1e6bff4ec480.slice/crio-8cb3d07b6b8cecf3d2359350b721be2b28b751752ecde66cde42493dc95817ed WatchSource:0}: Error finding container 8cb3d07b6b8cecf3d2359350b721be2b28b751752ecde66cde42493dc95817ed: Status 404 returned error can't find the container with id 8cb3d07b6b8cecf3d2359350b721be2b28b751752ecde66cde42493dc95817ed Jul 13 22:30:00.408524 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.408501 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733030-jql45"] Jul 13 22:30:00.420436 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:30:00.420413 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd885b15c_833a_4905_98c5_df9c702ed461.slice/crio-5a1269e9c2e242cbb38e1acf6767747ed1d5be4c8aa6ebdf06401619b0c898d5 WatchSource:0}: Error finding container 5a1269e9c2e242cbb38e1acf6767747ed1d5be4c8aa6ebdf06401619b0c898d5: Status 404 returned error can't find the container with id 5a1269e9c2e242cbb38e1acf6767747ed1d5be4c8aa6ebdf06401619b0c898d5 Jul 13 22:30:00.432802 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.432772 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61b7b8-29733030-57z4c"] Jul 13 22:30:00.436605 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:30:00.436580 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c7db64_1dbe_4d45_87b5_51bb850bf864.slice/crio-9c5202ef6527f53f8533a816839c7f13a3538d305b950c4dd244d56e1c5a24f9 WatchSource:0}: Error finding container 9c5202ef6527f53f8533a816839c7f13a3538d305b950c4dd244d56e1c5a24f9: Status 404 returned error can't find the container with id 9c5202ef6527f53f8533a816839c7f13a3538d305b950c4dd244d56e1c5a24f9 Jul 13 22:30:00.689853 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.689826 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd15db-29733030-q2xws" event={"ID":"cc81fe52-6827-4fbd-a2bd-1e6bff4ec480","Type":"ContainerStarted","Data":"8cb3d07b6b8cecf3d2359350b721be2b28b751752ecde66cde42493dc95817ed"} Jul 13 22:30:00.691478 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.691446 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61b7b8-29733030-57z4c" event={"ID":"27c7db64-1dbe-4d45-87b5-51bb850bf864","Type":"ContainerStarted","Data":"869a16e033fdf485867556d92307e622b7bd58125522314bf04ec074a6854fae"} Jul 13 22:30:00.691612 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.691502 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61b7b8-29733030-57z4c" event={"ID":"27c7db64-1dbe-4d45-87b5-51bb850bf864","Type":"ContainerStarted","Data":"9c5202ef6527f53f8533a816839c7f13a3538d305b950c4dd244d56e1c5a24f9"} Jul 13 22:30:00.692754 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.692733 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733030-jql45" event={"ID":"d885b15c-833a-4905-98c5-df9c702ed461","Type":"ContainerStarted","Data":"5a1269e9c2e242cbb38e1acf6767747ed1d5be4c8aa6ebdf06401619b0c898d5"} Jul 13 22:30:00.704043 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:00.703997 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61b7b8-29733030-57z4c" podStartSLOduration=0.703978627 podStartE2EDuration="703.978627ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:30:00.703311817 +0000 UTC m=+1913.360516591" watchObservedRunningTime="2026-07-13 22:30:00.703978627 +0000 UTC m=+1913.361183398" Jul 13 22:30:01.699818 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:01.699770 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733030-jql45" event={"ID":"d885b15c-833a-4905-98c5-df9c702ed461","Type":"ContainerStarted","Data":"44f485d016e5f6f8351b66b3bb8b589d9977a4a9b9eba89d71f6dc5f58bf295e"} Jul 13 22:30:01.701441 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:01.701414 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd15db-29733030-q2xws" event={"ID":"cc81fe52-6827-4fbd-a2bd-1e6bff4ec480","Type":"ContainerStarted","Data":"b369007214310794374670e989baa3babf31aa69cf8c270c7f6a9c107f410949"} Jul 13 22:30:01.714021 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:01.713967 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733030-jql45" podStartSLOduration=1.7139543449999999 podStartE2EDuration="1.713954345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:30:01.713418747 +0000 UTC m=+1914.370623511" watchObservedRunningTime="2026-07-13 22:30:01.713954345 +0000 UTC m=+1914.371159118" Jul 13 22:30:01.728519 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:01.728472 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd15db-29733030-q2xws" podStartSLOduration=1.7284589289999999 podStartE2EDuration="1.728458929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:30:01.726686083 +0000 UTC m=+1914.383890860" watchObservedRunningTime="2026-07-13 22:30:01.728458929 +0000 UTC m=+1914.385663697" Jul 13 22:30:07.669722 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:07.669683 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w"] Jul 13 22:30:07.670688 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:07.670617 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" podUID="8e3d6513-5ae2-4684-be2a-33824317157c" containerName="maas-api" containerID="cri-o://4cf3dd81c3baa9c8695449f299fbf32468fae20e85713e7ca15c001e8cbb22fd" gracePeriod=30 Jul 13 22:30:07.691445 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:07.691412 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd15db-29733030-q2xws"] Jul 13 22:30:07.691615 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:07.691594 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd15db-29733030-q2xws" podUID="cc81fe52-6827-4fbd-a2bd-1e6bff4ec480" containerName="cleanup" containerID="cri-o://b369007214310794374670e989baa3babf31aa69cf8c270c7f6a9c107f410949" gracePeriod=30 Jul 13 22:30:08.131182 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:08.131154 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" Jul 13 22:30:08.288114 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:08.288017 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mxh2\" (UniqueName: \"kubernetes.io/projected/8e3d6513-5ae2-4684-be2a-33824317157c-kube-api-access-4mxh2\") pod \"8e3d6513-5ae2-4684-be2a-33824317157c\" (UID: \"8e3d6513-5ae2-4684-be2a-33824317157c\") " Jul 13 22:30:08.288114 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:08.288057 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e3d6513-5ae2-4684-be2a-33824317157c-maas-api-tls\") pod \"8e3d6513-5ae2-4684-be2a-33824317157c\" (UID: \"8e3d6513-5ae2-4684-be2a-33824317157c\") " Jul 13 22:30:08.288114 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:08.288093 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e3d6513-5ae2-4684-be2a-33824317157c-openshift-service-ca\") pod \"8e3d6513-5ae2-4684-be2a-33824317157c\" (UID: \"8e3d6513-5ae2-4684-be2a-33824317157c\") " Jul 13 22:30:08.288575 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:08.288523 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3d6513-5ae2-4684-be2a-33824317157c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8e3d6513-5ae2-4684-be2a-33824317157c" (UID: "8e3d6513-5ae2-4684-be2a-33824317157c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:30:08.290266 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:08.290242 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3d6513-5ae2-4684-be2a-33824317157c-kube-api-access-4mxh2" (OuterVolumeSpecName: "kube-api-access-4mxh2") pod "8e3d6513-5ae2-4684-be2a-33824317157c" (UID: "8e3d6513-5ae2-4684-be2a-33824317157c"). InnerVolumeSpecName "kube-api-access-4mxh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:30:08.290345 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:08.290285 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e3d6513-5ae2-4684-be2a-33824317157c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8e3d6513-5ae2-4684-be2a-33824317157c" (UID: "8e3d6513-5ae2-4684-be2a-33824317157c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:30:08.389578 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:08.389532 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mxh2\" (UniqueName: \"kubernetes.io/projected/8e3d6513-5ae2-4684-be2a-33824317157c-kube-api-access-4mxh2\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:30:08.389578 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:08.389572 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e3d6513-5ae2-4684-be2a-33824317157c-maas-api-tls\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:30:08.389578 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:08.389582 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e3d6513-5ae2-4684-be2a-33824317157c-openshift-service-ca\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:30:08.739606 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:08.739563 2569 generic.go:356] "Generic (PLEG): container finished" podID="8e3d6513-5ae2-4684-be2a-33824317157c" containerID="4cf3dd81c3baa9c8695449f299fbf32468fae20e85713e7ca15c001e8cbb22fd" exitCode=0 Jul 13 22:30:08.740020 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:08.739627 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" Jul 13 22:30:08.740020 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:08.739630 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" event={"ID":"8e3d6513-5ae2-4684-be2a-33824317157c","Type":"ContainerDied","Data":"4cf3dd81c3baa9c8695449f299fbf32468fae20e85713e7ca15c001e8cbb22fd"} Jul 13 22:30:08.740020 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:08.739713 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w" event={"ID":"8e3d6513-5ae2-4684-be2a-33824317157c","Type":"ContainerDied","Data":"ad9dc027646dedd48097534990aeb06ddadd92d514c3a331f17b4723049e5a64"} Jul 13 22:30:08.740020 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:08.739737 2569 scope.go:117] "RemoveContainer" containerID="4cf3dd81c3baa9c8695449f299fbf32468fae20e85713e7ca15c001e8cbb22fd" Jul 13 22:30:08.749287 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:08.749268 2569 scope.go:117] "RemoveContainer" containerID="4cf3dd81c3baa9c8695449f299fbf32468fae20e85713e7ca15c001e8cbb22fd" Jul 13 22:30:08.749529 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:30:08.749513 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf3dd81c3baa9c8695449f299fbf32468fae20e85713e7ca15c001e8cbb22fd\": container with ID starting with 4cf3dd81c3baa9c8695449f299fbf32468fae20e85713e7ca15c001e8cbb22fd not found: ID does not exist" containerID="4cf3dd81c3baa9c8695449f299fbf32468fae20e85713e7ca15c001e8cbb22fd" Jul 13 22:30:08.749580 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:08.749556 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf3dd81c3baa9c8695449f299fbf32468fae20e85713e7ca15c001e8cbb22fd"} err="failed to get container status \"4cf3dd81c3baa9c8695449f299fbf32468fae20e85713e7ca15c001e8cbb22fd\": rpc error: code = NotFound desc = could not find container \"4cf3dd81c3baa9c8695449f299fbf32468fae20e85713e7ca15c001e8cbb22fd\": container with ID starting with 4cf3dd81c3baa9c8695449f299fbf32468fae20e85713e7ca15c001e8cbb22fd not found: ID does not exist" Jul 13 22:30:08.761703 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:08.761676 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w"] Jul 13 22:30:08.763564 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:08.763529 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd15db-5dc8b66f6d-5hs4w"] Jul 13 22:30:09.931101 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:09.931068 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e3d6513-5ae2-4684-be2a-33824317157c" path="/var/lib/kubelet/pods/8e3d6513-5ae2-4684-be2a-33824317157c/volumes" Jul 13 22:30:37.892380 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:37.892350 2569 generic.go:356] "Generic (PLEG): container finished" podID="cc81fe52-6827-4fbd-a2bd-1e6bff4ec480" containerID="b369007214310794374670e989baa3babf31aa69cf8c270c7f6a9c107f410949" exitCode=137 Jul 13 22:30:37.892759 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:37.892390 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd15db-29733030-q2xws" event={"ID":"cc81fe52-6827-4fbd-a2bd-1e6bff4ec480","Type":"ContainerDied","Data":"b369007214310794374670e989baa3babf31aa69cf8c270c7f6a9c107f410949"} Jul 13 22:30:37.892759 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:37.892426 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd15db-29733030-q2xws" event={"ID":"cc81fe52-6827-4fbd-a2bd-1e6bff4ec480","Type":"ContainerDied","Data":"8cb3d07b6b8cecf3d2359350b721be2b28b751752ecde66cde42493dc95817ed"} Jul 13 22:30:37.892759 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:37.892443 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cb3d07b6b8cecf3d2359350b721be2b28b751752ecde66cde42493dc95817ed" Jul 13 22:30:37.893211 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:37.893182 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd15db-29733030-q2xws" Jul 13 22:30:38.897170 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:38.897139 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd15db-29733030-q2xws" Jul 13 22:30:38.913099 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:38.913067 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd15db-29733030-q2xws"] Jul 13 22:30:38.918483 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:38.918445 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fd15db-29733030-q2xws"] Jul 13 22:30:39.931200 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:39.931162 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc81fe52-6827-4fbd-a2bd-1e6bff4ec480" path="/var/lib/kubelet/pods/cc81fe52-6827-4fbd-a2bd-1e6bff4ec480/volumes" Jul 13 22:30:49.415286 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:49.415248 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g"] Jul 13 22:30:49.415834 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:49.415505 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" podUID="92929369-5eeb-46d4-9ba7-5c51b56c9f33" containerName="istio-proxy" containerID="cri-o://3f1f3e501096ca838696af637507efd8c421ed732fab3d6ca56a19f8091fcb42" gracePeriod=30 Jul 13 22:30:50.478858 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.478822 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc"] Jul 13 22:30:50.479627 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.479595 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" podUID="8b7fc6cf-9e57-493b-987a-963ee5681ae5" containerName="maas-api" containerID="cri-o://7528848b05901cb945cb22db7428637b54a22be01a0b26e08d69d991867c2f10" gracePeriod=30 Jul 13 22:30:50.511419 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.511393 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61b7b8-29733030-57z4c"] Jul 13 22:30:50.511624 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.511598 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61b7b8-29733030-57z4c" podUID="27c7db64-1dbe-4d45-87b5-51bb850bf864" containerName="cleanup" containerID="cri-o://869a16e033fdf485867556d92307e622b7bd58125522314bf04ec074a6854fae" gracePeriod=30 Jul 13 22:30:50.741445 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.741373 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" Jul 13 22:30:50.892017 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.891980 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b7fc6cf-9e57-493b-987a-963ee5681ae5-maas-api-tls\") pod \"8b7fc6cf-9e57-493b-987a-963ee5681ae5\" (UID: \"8b7fc6cf-9e57-493b-987a-963ee5681ae5\") " Jul 13 22:30:50.892188 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.892035 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kclxq\" (UniqueName: \"kubernetes.io/projected/8b7fc6cf-9e57-493b-987a-963ee5681ae5-kube-api-access-kclxq\") pod \"8b7fc6cf-9e57-493b-987a-963ee5681ae5\" (UID: \"8b7fc6cf-9e57-493b-987a-963ee5681ae5\") " Jul 13 22:30:50.892188 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.892097 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7fc6cf-9e57-493b-987a-963ee5681ae5-openshift-service-ca\") pod \"8b7fc6cf-9e57-493b-987a-963ee5681ae5\" (UID: \"8b7fc6cf-9e57-493b-987a-963ee5681ae5\") " Jul 13 22:30:50.892527 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.892489 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7fc6cf-9e57-493b-987a-963ee5681ae5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8b7fc6cf-9e57-493b-987a-963ee5681ae5" (UID: "8b7fc6cf-9e57-493b-987a-963ee5681ae5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:30:50.894030 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.894006 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7fc6cf-9e57-493b-987a-963ee5681ae5-kube-api-access-kclxq" (OuterVolumeSpecName: "kube-api-access-kclxq") pod "8b7fc6cf-9e57-493b-987a-963ee5681ae5" (UID: "8b7fc6cf-9e57-493b-987a-963ee5681ae5"). InnerVolumeSpecName "kube-api-access-kclxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:30:50.894114 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.894095 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7fc6cf-9e57-493b-987a-963ee5681ae5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8b7fc6cf-9e57-493b-987a-963ee5681ae5" (UID: "8b7fc6cf-9e57-493b-987a-963ee5681ae5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:30:50.960803 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.960775 2569 generic.go:356] "Generic (PLEG): container finished" podID="8b7fc6cf-9e57-493b-987a-963ee5681ae5" containerID="7528848b05901cb945cb22db7428637b54a22be01a0b26e08d69d991867c2f10" exitCode=0 Jul 13 22:30:50.960935 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.960811 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" Jul 13 22:30:50.960935 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.960861 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" event={"ID":"8b7fc6cf-9e57-493b-987a-963ee5681ae5","Type":"ContainerDied","Data":"7528848b05901cb945cb22db7428637b54a22be01a0b26e08d69d991867c2f10"} Jul 13 22:30:50.960935 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.960909 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc" event={"ID":"8b7fc6cf-9e57-493b-987a-963ee5681ae5","Type":"ContainerDied","Data":"c76e404392b66f3aa9a0a15216402a40febf26f2466f41faaff2e78d34fc63b7"} Jul 13 22:30:50.961078 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.960936 2569 scope.go:117] "RemoveContainer" containerID="7528848b05901cb945cb22db7428637b54a22be01a0b26e08d69d991867c2f10" Jul 13 22:30:50.969694 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.969677 2569 scope.go:117] "RemoveContainer" containerID="7528848b05901cb945cb22db7428637b54a22be01a0b26e08d69d991867c2f10" Jul 13 22:30:50.969919 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:30:50.969901 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7528848b05901cb945cb22db7428637b54a22be01a0b26e08d69d991867c2f10\": container with ID starting with 7528848b05901cb945cb22db7428637b54a22be01a0b26e08d69d991867c2f10 not found: ID does not exist" containerID="7528848b05901cb945cb22db7428637b54a22be01a0b26e08d69d991867c2f10" Jul 13 22:30:50.969962 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.969927 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7528848b05901cb945cb22db7428637b54a22be01a0b26e08d69d991867c2f10"} err="failed to get container status \"7528848b05901cb945cb22db7428637b54a22be01a0b26e08d69d991867c2f10\": rpc error: code = NotFound desc = could not find container \"7528848b05901cb945cb22db7428637b54a22be01a0b26e08d69d991867c2f10\": container with ID starting with 7528848b05901cb945cb22db7428637b54a22be01a0b26e08d69d991867c2f10 not found: ID does not exist" Jul 13 22:30:50.987215 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.987192 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc"] Jul 13 22:30:50.989099 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.989075 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61b7b8-6cbdcffc5f-nxdcc"] Jul 13 22:30:50.993124 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.993071 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7fc6cf-9e57-493b-987a-963ee5681ae5-openshift-service-ca\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:30:50.993124 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.993092 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b7fc6cf-9e57-493b-987a-963ee5681ae5-maas-api-tls\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:30:50.993124 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:50.993102 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kclxq\" (UniqueName: \"kubernetes.io/projected/8b7fc6cf-9e57-493b-987a-963ee5681ae5-kube-api-access-kclxq\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:30:51.931273 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:51.931243 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b7fc6cf-9e57-493b-987a-963ee5681ae5" path="/var/lib/kubelet/pods/8b7fc6cf-9e57-493b-987a-963ee5681ae5/volumes" Jul 13 22:30:54.671850 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.671828 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:30:54.826891 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.826861 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-workload-socket\") pod \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " Jul 13 22:30:54.827089 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.826916 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-token\") pod \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " Jul 13 22:30:54.827089 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.826950 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bslkq\" (UniqueName: \"kubernetes.io/projected/92929369-5eeb-46d4-9ba7-5c51b56c9f33-kube-api-access-bslkq\") pod \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " Jul 13 22:30:54.827089 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.826976 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-podinfo\") pod \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " Jul 13 22:30:54.827089 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.827047 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-workload-certs\") pod \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " Jul 13 22:30:54.827319 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.827090 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-envoy\") pod \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " Jul 13 22:30:54.827319 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.827118 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istiod-ca-cert\") pod \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " Jul 13 22:30:54.827319 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.827124 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "92929369-5eeb-46d4-9ba7-5c51b56c9f33" (UID: "92929369-5eeb-46d4-9ba7-5c51b56c9f33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:30:54.827319 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.827156 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-data\") pod \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " Jul 13 22:30:54.827319 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.827181 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-credential-socket\") pod \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\" (UID: \"92929369-5eeb-46d4-9ba7-5c51b56c9f33\") " Jul 13 22:30:54.827607 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.827335 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "92929369-5eeb-46d4-9ba7-5c51b56c9f33" (UID: "92929369-5eeb-46d4-9ba7-5c51b56c9f33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:30:54.827669 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.827602 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-workload-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:30:54.827669 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.827624 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-workload-certs\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:30:54.827777 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.827678 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "92929369-5eeb-46d4-9ba7-5c51b56c9f33" (UID: "92929369-5eeb-46d4-9ba7-5c51b56c9f33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:30:54.827831 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.827793 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "92929369-5eeb-46d4-9ba7-5c51b56c9f33" (UID: "92929369-5eeb-46d4-9ba7-5c51b56c9f33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:30:54.828253 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.828224 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-data" (OuterVolumeSpecName: "istio-data") pod "92929369-5eeb-46d4-9ba7-5c51b56c9f33" (UID: "92929369-5eeb-46d4-9ba7-5c51b56c9f33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:30:54.829283 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.829258 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "92929369-5eeb-46d4-9ba7-5c51b56c9f33" (UID: "92929369-5eeb-46d4-9ba7-5c51b56c9f33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:30:54.829429 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.829304 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92929369-5eeb-46d4-9ba7-5c51b56c9f33-kube-api-access-bslkq" (OuterVolumeSpecName: "kube-api-access-bslkq") pod "92929369-5eeb-46d4-9ba7-5c51b56c9f33" (UID: "92929369-5eeb-46d4-9ba7-5c51b56c9f33"). InnerVolumeSpecName "kube-api-access-bslkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:30:54.829429 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.829365 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "92929369-5eeb-46d4-9ba7-5c51b56c9f33" (UID: "92929369-5eeb-46d4-9ba7-5c51b56c9f33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:30:54.829429 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.829375 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-token" (OuterVolumeSpecName: "istio-token") pod "92929369-5eeb-46d4-9ba7-5c51b56c9f33" (UID: "92929369-5eeb-46d4-9ba7-5c51b56c9f33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:30:54.928281 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.928258 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-envoy\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:30:54.928281 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.928279 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istiod-ca-cert\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:30:54.928419 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.928288 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-data\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:30:54.928419 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.928296 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92929369-5eeb-46d4-9ba7-5c51b56c9f33-credential-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:30:54.928419 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.928304 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-token\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:30:54.928419 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.928312 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bslkq\" (UniqueName: \"kubernetes.io/projected/92929369-5eeb-46d4-9ba7-5c51b56c9f33-kube-api-access-bslkq\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:30:54.928419 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.928320 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92929369-5eeb-46d4-9ba7-5c51b56c9f33-istio-podinfo\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:30:54.984490 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.984462 2569 generic.go:356] "Generic (PLEG): container finished" podID="92929369-5eeb-46d4-9ba7-5c51b56c9f33" containerID="3f1f3e501096ca838696af637507efd8c421ed732fab3d6ca56a19f8091fcb42" exitCode=0 Jul 13 22:30:54.984623 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.984506 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" Jul 13 22:30:54.984623 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.984560 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" event={"ID":"92929369-5eeb-46d4-9ba7-5c51b56c9f33","Type":"ContainerDied","Data":"3f1f3e501096ca838696af637507efd8c421ed732fab3d6ca56a19f8091fcb42"} Jul 13 22:30:54.984623 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.984608 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g" event={"ID":"92929369-5eeb-46d4-9ba7-5c51b56c9f33","Type":"ContainerDied","Data":"6aa4c9003fd162dfefaf44b52cd2fb0f060b778e03cfd16fe0e99eed3e53439f"} Jul 13 22:30:54.984780 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.984636 2569 scope.go:117] "RemoveContainer" containerID="3f1f3e501096ca838696af637507efd8c421ed732fab3d6ca56a19f8091fcb42" Jul 13 22:30:54.993979 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:54.993961 2569 scope.go:117] "RemoveContainer" containerID="259ab7fe616399913daf162707e0c0bdf7a220d56a07342b927c228eb90a4500" Jul 13 22:30:55.001836 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:55.001817 2569 scope.go:117] "RemoveContainer" containerID="3f1f3e501096ca838696af637507efd8c421ed732fab3d6ca56a19f8091fcb42" Jul 13 22:30:55.002070 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:30:55.002051 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f1f3e501096ca838696af637507efd8c421ed732fab3d6ca56a19f8091fcb42\": container with ID starting with 3f1f3e501096ca838696af637507efd8c421ed732fab3d6ca56a19f8091fcb42 not found: ID does not exist" containerID="3f1f3e501096ca838696af637507efd8c421ed732fab3d6ca56a19f8091fcb42" Jul 13 22:30:55.002124 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:55.002078 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f1f3e501096ca838696af637507efd8c421ed732fab3d6ca56a19f8091fcb42"} err="failed to get container status \"3f1f3e501096ca838696af637507efd8c421ed732fab3d6ca56a19f8091fcb42\": rpc error: code = NotFound desc = could not find container \"3f1f3e501096ca838696af637507efd8c421ed732fab3d6ca56a19f8091fcb42\": container with ID starting with 3f1f3e501096ca838696af637507efd8c421ed732fab3d6ca56a19f8091fcb42 not found: ID does not exist" Jul 13 22:30:55.002124 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:55.002094 2569 scope.go:117] "RemoveContainer" containerID="259ab7fe616399913daf162707e0c0bdf7a220d56a07342b927c228eb90a4500" Jul 13 22:30:55.002354 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:30:55.002334 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"259ab7fe616399913daf162707e0c0bdf7a220d56a07342b927c228eb90a4500\": container with ID starting with 259ab7fe616399913daf162707e0c0bdf7a220d56a07342b927c228eb90a4500 not found: ID does not exist" containerID="259ab7fe616399913daf162707e0c0bdf7a220d56a07342b927c228eb90a4500" Jul 13 22:30:55.002403 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:55.002361 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"259ab7fe616399913daf162707e0c0bdf7a220d56a07342b927c228eb90a4500"} err="failed to get container status \"259ab7fe616399913daf162707e0c0bdf7a220d56a07342b927c228eb90a4500\": rpc error: code = NotFound desc = could not find container \"259ab7fe616399913daf162707e0c0bdf7a220d56a07342b927c228eb90a4500\": container with ID starting with 259ab7fe616399913daf162707e0c0bdf7a220d56a07342b927c228eb90a4500 not found: ID does not exist" Jul 13 22:30:55.006846 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:55.006827 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g"] Jul 13 22:30:55.011075 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:55.011056 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fd15db-openshift-default-5474558869-cdd8g"] Jul 13 22:30:55.931287 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:55.931250 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92929369-5eeb-46d4-9ba7-5c51b56c9f33" path="/var/lib/kubelet/pods/92929369-5eeb-46d4-9ba7-5c51b56c9f33/volumes" Jul 13 22:30:56.459264 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:56.459232 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx"] Jul 13 22:30:56.459497 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:56.459470 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" podUID="405aa82e-1358-4ffc-8653-eccb3956c435" containerName="istio-proxy" containerID="cri-o://541fd371e256da5fa2a04b47cdf00d6f69a0984b028e5dd6dcc1433ecc407cd7" gracePeriod=30 Jul 13 22:30:57.339380 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.339346 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4"] Jul 13 22:30:57.339993 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.339962 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b7fc6cf-9e57-493b-987a-963ee5681ae5" containerName="maas-api" Jul 13 22:30:57.339993 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.339990 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b7fc6cf-9e57-493b-987a-963ee5681ae5" containerName="maas-api" Jul 13 22:30:57.340174 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.340015 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92929369-5eeb-46d4-9ba7-5c51b56c9f33" containerName="istio-proxy" Jul 13 22:30:57.340174 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.340025 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="92929369-5eeb-46d4-9ba7-5c51b56c9f33" containerName="istio-proxy" Jul 13 22:30:57.340174 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.340037 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92929369-5eeb-46d4-9ba7-5c51b56c9f33" containerName="istio-proxy" Jul 13 22:30:57.340174 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.340046 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="92929369-5eeb-46d4-9ba7-5c51b56c9f33" containerName="istio-proxy" Jul 13 22:30:57.340174 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.340060 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e3d6513-5ae2-4684-be2a-33824317157c" containerName="maas-api" Jul 13 22:30:57.340174 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.340069 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e3d6513-5ae2-4684-be2a-33824317157c" containerName="maas-api" Jul 13 22:30:57.340174 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.340077 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc81fe52-6827-4fbd-a2bd-1e6bff4ec480" containerName="cleanup" Jul 13 22:30:57.340174 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.340085 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc81fe52-6827-4fbd-a2bd-1e6bff4ec480" containerName="cleanup" Jul 13 22:30:57.340486 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.340185 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b7fc6cf-9e57-493b-987a-963ee5681ae5" containerName="maas-api" Jul 13 22:30:57.340486 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.340197 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e3d6513-5ae2-4684-be2a-33824317157c" containerName="maas-api" Jul 13 22:30:57.340486 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.340204 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="92929369-5eeb-46d4-9ba7-5c51b56c9f33" containerName="istio-proxy" Jul 13 22:30:57.340486 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.340211 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="92929369-5eeb-46d4-9ba7-5c51b56c9f33" containerName="istio-proxy" Jul 13 22:30:57.340486 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.340218 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc81fe52-6827-4fbd-a2bd-1e6bff4ec480" containerName="cleanup" Jul 13 22:30:57.344902 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.344883 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.347573 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.347553 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-232fa0-openshift-default-dockercfg-jwr9f\"" Jul 13 22:30:57.355850 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.355827 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4"] Jul 13 22:30:57.448591 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.448517 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kzd9\" (UniqueName: \"kubernetes.io/projected/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-kube-api-access-6kzd9\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.448796 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.448624 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-envoy\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.448796 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.448650 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istiod-ca-cert\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.448796 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.448717 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-token\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.448796 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.448742 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-credential-socket\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.448796 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.448769 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-workload-certs\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.448979 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.448801 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-podinfo\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.448979 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.448875 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-workload-socket\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.448979 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.448919 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-data\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.549793 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.549749 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-workload-socket\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.549793 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.549804 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-data\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.550044 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.549831 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kzd9\" (UniqueName: \"kubernetes.io/projected/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-kube-api-access-6kzd9\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.550044 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.549858 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-envoy\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.550044 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.549873 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istiod-ca-cert\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.550044 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.549921 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-token\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.550044 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.549954 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-credential-socket\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.550044 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.549981 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-workload-certs\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.550044 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.550014 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-podinfo\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.550412 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.550176 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-workload-socket\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.550497 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.550481 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-data\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.550582 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.550515 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-workload-certs\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.550582 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.550563 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-credential-socket\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.550660 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.550637 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istiod-ca-cert\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.552237 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.552218 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-envoy\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.552528 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.552508 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-podinfo\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.561971 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.561949 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-token\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.562090 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.562077 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kzd9\" (UniqueName: \"kubernetes.io/projected/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-kube-api-access-6kzd9\") pod \"e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.656722 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.656657 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:57.808228 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.808204 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4"] Jul 13 22:30:57.810312 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:30:57.810282 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27058e4c_0e92_4347_96d7_3ffa6a52ac9e.slice/crio-00dc692a8e86ebcbc1eeaee48efa8514bcc89ea194672aa4b16c69902e90e9fb WatchSource:0}: Error finding container 00dc692a8e86ebcbc1eeaee48efa8514bcc89ea194672aa4b16c69902e90e9fb: Status 404 returned error can't find the container with id 00dc692a8e86ebcbc1eeaee48efa8514bcc89ea194672aa4b16c69902e90e9fb Jul 13 22:30:57.812463 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.812433 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:30:57.812565 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.812497 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:30:57.812565 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:57.812525 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:30:58.002295 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:58.002252 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" event={"ID":"27058e4c-0e92-4347-96d7-3ffa6a52ac9e","Type":"ContainerStarted","Data":"aa103942da049c91ef50360405e3bc074545605308bb8ff80e1152425765e077"} Jul 13 22:30:58.002496 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:58.002323 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" event={"ID":"27058e4c-0e92-4347-96d7-3ffa6a52ac9e","Type":"ContainerStarted","Data":"00dc692a8e86ebcbc1eeaee48efa8514bcc89ea194672aa4b16c69902e90e9fb"} Jul 13 22:30:58.025624 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:58.025578 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" podStartSLOduration=1.025565237 podStartE2EDuration="1.025565237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:30:58.022593148 +0000 UTC m=+1970.679797913" watchObservedRunningTime="2026-07-13 22:30:58.025565237 +0000 UTC m=+1970.682770002" Jul 13 22:30:58.657644 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:58.657612 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:58.662370 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:58.662347 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:59.008686 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:59.008655 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:59.009804 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:59.009781 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:30:59.998983 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:30:59.998953 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm"] Jul 13 22:31:00.002851 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:00.002826 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" Jul 13 22:31:00.010174 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:00.010143 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-232fa0-serving-cert\"" Jul 13 22:31:00.021707 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:00.021669 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm"] Jul 13 22:31:00.024563 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:00.024494 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733030-jql45"] Jul 13 22:31:00.024740 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:00.024712 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733030-jql45" podUID="d885b15c-833a-4905-98c5-df9c702ed461" containerName="cleanup" containerID="cri-o://44f485d016e5f6f8351b66b3bb8b589d9977a4a9b9eba89d71f6dc5f58bf295e" gracePeriod=30 Jul 13 22:31:00.176968 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:00.176924 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f59f22b-774f-4d69-85fe-65494b6bb12b-openshift-service-ca\") pod \"maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm\" (UID: \"9f59f22b-774f-4d69-85fe-65494b6bb12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" Jul 13 22:31:00.176968 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:00.176965 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm8sx\" (UniqueName: \"kubernetes.io/projected/9f59f22b-774f-4d69-85fe-65494b6bb12b-kube-api-access-wm8sx\") pod \"maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm\" (UID: \"9f59f22b-774f-4d69-85fe-65494b6bb12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" Jul 13 22:31:00.177150 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:00.177081 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f59f22b-774f-4d69-85fe-65494b6bb12b-maas-api-tls\") pod \"maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm\" (UID: \"9f59f22b-774f-4d69-85fe-65494b6bb12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" Jul 13 22:31:00.278365 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:00.278284 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f59f22b-774f-4d69-85fe-65494b6bb12b-maas-api-tls\") pod \"maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm\" (UID: \"9f59f22b-774f-4d69-85fe-65494b6bb12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" Jul 13 22:31:00.278523 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:31:00.278409 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-serving-cert: secret "maas-api-e2e-api-a-232fa0-serving-cert" not found Jul 13 22:31:00.278523 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:00.278428 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f59f22b-774f-4d69-85fe-65494b6bb12b-openshift-service-ca\") pod \"maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm\" (UID: \"9f59f22b-774f-4d69-85fe-65494b6bb12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" Jul 13 22:31:00.278523 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:00.278459 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm8sx\" (UniqueName: \"kubernetes.io/projected/9f59f22b-774f-4d69-85fe-65494b6bb12b-kube-api-access-wm8sx\") pod \"maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm\" (UID: \"9f59f22b-774f-4d69-85fe-65494b6bb12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" Jul 13 22:31:00.278523 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:31:00.278467 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f59f22b-774f-4d69-85fe-65494b6bb12b-maas-api-tls podName:9f59f22b-774f-4d69-85fe-65494b6bb12b nodeName:}" failed. No retries permitted until 2026-07-13 22:31:00.778451894 +0000 UTC m=+1973.435656641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f59f22b-774f-4d69-85fe-65494b6bb12b-maas-api-tls") pod "maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" (UID: "9f59f22b-774f-4d69-85fe-65494b6bb12b") : secret "maas-api-e2e-api-a-232fa0-serving-cert" not found Jul 13 22:31:00.279061 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:00.279042 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f59f22b-774f-4d69-85fe-65494b6bb12b-openshift-service-ca\") pod \"maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm\" (UID: \"9f59f22b-774f-4d69-85fe-65494b6bb12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" Jul 13 22:31:00.288171 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:00.288146 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm8sx\" (UniqueName: \"kubernetes.io/projected/9f59f22b-774f-4d69-85fe-65494b6bb12b-kube-api-access-wm8sx\") pod \"maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm\" (UID: \"9f59f22b-774f-4d69-85fe-65494b6bb12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" Jul 13 22:31:00.783671 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:00.783640 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f59f22b-774f-4d69-85fe-65494b6bb12b-maas-api-tls\") pod \"maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm\" (UID: \"9f59f22b-774f-4d69-85fe-65494b6bb12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" Jul 13 22:31:00.786057 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:00.786025 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f59f22b-774f-4d69-85fe-65494b6bb12b-maas-api-tls\") pod \"maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm\" (UID: \"9f59f22b-774f-4d69-85fe-65494b6bb12b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" Jul 13 22:31:00.914918 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:00.914891 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" Jul 13 22:31:01.067381 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.067356 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm"] Jul 13 22:31:01.069887 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:31:01.069857 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f59f22b_774f_4d69_85fe_65494b6bb12b.slice/crio-1c8d790758f11b53cb748c184617b1de98f9180fb234fc2094360f7b01655fa9 WatchSource:0}: Error finding container 1c8d790758f11b53cb748c184617b1de98f9180fb234fc2094360f7b01655fa9: Status 404 returned error can't find the container with id 1c8d790758f11b53cb748c184617b1de98f9180fb234fc2094360f7b01655fa9 Jul 13 22:31:01.731213 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.731154 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:31:01.796629 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.796597 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405aa82e-1358-4ffc-8653-eccb3956c435-istio-token\") pod \"405aa82e-1358-4ffc-8653-eccb3956c435\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " Jul 13 22:31:01.796793 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.796641 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-credential-socket\") pod \"405aa82e-1358-4ffc-8653-eccb3956c435\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " Jul 13 22:31:01.796793 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.796676 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-workload-certs\") pod \"405aa82e-1358-4ffc-8653-eccb3956c435\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " Jul 13 22:31:01.796793 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.796702 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7clzv\" (UniqueName: \"kubernetes.io/projected/405aa82e-1358-4ffc-8653-eccb3956c435-kube-api-access-7clzv\") pod \"405aa82e-1358-4ffc-8653-eccb3956c435\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " Jul 13 22:31:01.796793 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.796762 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405aa82e-1358-4ffc-8653-eccb3956c435-istio-podinfo\") pod \"405aa82e-1358-4ffc-8653-eccb3956c435\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " Jul 13 22:31:01.797009 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.796818 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405aa82e-1358-4ffc-8653-eccb3956c435-istiod-ca-cert\") pod \"405aa82e-1358-4ffc-8653-eccb3956c435\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " Jul 13 22:31:01.797009 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.796848 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-workload-socket\") pod \"405aa82e-1358-4ffc-8653-eccb3956c435\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " Jul 13 22:31:01.797009 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.796895 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-istio-envoy\") pod \"405aa82e-1358-4ffc-8653-eccb3956c435\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " Jul 13 22:31:01.797009 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.796932 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-istio-data\") pod \"405aa82e-1358-4ffc-8653-eccb3956c435\" (UID: \"405aa82e-1358-4ffc-8653-eccb3956c435\") " Jul 13 22:31:01.797009 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.796938 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "405aa82e-1358-4ffc-8653-eccb3956c435" (UID: "405aa82e-1358-4ffc-8653-eccb3956c435"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:31:01.797009 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.796965 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "405aa82e-1358-4ffc-8653-eccb3956c435" (UID: "405aa82e-1358-4ffc-8653-eccb3956c435"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:31:01.797300 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.797123 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "405aa82e-1358-4ffc-8653-eccb3956c435" (UID: "405aa82e-1358-4ffc-8653-eccb3956c435"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:31:01.797300 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.797241 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-workload-certs\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:01.797300 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.797262 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-workload-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:01.797300 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.797253 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405aa82e-1358-4ffc-8653-eccb3956c435-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "405aa82e-1358-4ffc-8653-eccb3956c435" (UID: "405aa82e-1358-4ffc-8653-eccb3956c435"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:31:01.797300 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.797276 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-credential-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:01.797852 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.797830 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-istio-data" (OuterVolumeSpecName: "istio-data") pod "405aa82e-1358-4ffc-8653-eccb3956c435" (UID: "405aa82e-1358-4ffc-8653-eccb3956c435"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:31:01.798843 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.798822 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405aa82e-1358-4ffc-8653-eccb3956c435-kube-api-access-7clzv" (OuterVolumeSpecName: "kube-api-access-7clzv") pod "405aa82e-1358-4ffc-8653-eccb3956c435" (UID: "405aa82e-1358-4ffc-8653-eccb3956c435"). InnerVolumeSpecName "kube-api-access-7clzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:31:01.799103 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.799088 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "405aa82e-1358-4ffc-8653-eccb3956c435" (UID: "405aa82e-1358-4ffc-8653-eccb3956c435"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:31:01.799186 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.799171 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/405aa82e-1358-4ffc-8653-eccb3956c435-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "405aa82e-1358-4ffc-8653-eccb3956c435" (UID: "405aa82e-1358-4ffc-8653-eccb3956c435"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:31:01.799237 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.799194 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405aa82e-1358-4ffc-8653-eccb3956c435-istio-token" (OuterVolumeSpecName: "istio-token") pod "405aa82e-1358-4ffc-8653-eccb3956c435" (UID: "405aa82e-1358-4ffc-8653-eccb3956c435"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:31:01.897814 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.897788 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-istio-envoy\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:01.897814 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.897812 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405aa82e-1358-4ffc-8653-eccb3956c435-istio-data\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:01.897991 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.897822 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405aa82e-1358-4ffc-8653-eccb3956c435-istio-token\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:01.897991 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.897832 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7clzv\" (UniqueName: \"kubernetes.io/projected/405aa82e-1358-4ffc-8653-eccb3956c435-kube-api-access-7clzv\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:01.897991 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.897841 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405aa82e-1358-4ffc-8653-eccb3956c435-istio-podinfo\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:01.897991 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:01.897850 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405aa82e-1358-4ffc-8653-eccb3956c435-istiod-ca-cert\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:02.030482 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:02.030387 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" event={"ID":"9f59f22b-774f-4d69-85fe-65494b6bb12b","Type":"ContainerStarted","Data":"5bfea4fe8b6e17e3a14923e25abf8e4705220d57d4386ba2cedf38b0de82f452"} Jul 13 22:31:02.030482 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:02.030454 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" event={"ID":"9f59f22b-774f-4d69-85fe-65494b6bb12b","Type":"ContainerStarted","Data":"1c8d790758f11b53cb748c184617b1de98f9180fb234fc2094360f7b01655fa9"} Jul 13 22:31:02.030680 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:02.030486 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" Jul 13 22:31:02.032080 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:02.032057 2569 generic.go:356] "Generic (PLEG): container finished" podID="405aa82e-1358-4ffc-8653-eccb3956c435" containerID="541fd371e256da5fa2a04b47cdf00d6f69a0984b028e5dd6dcc1433ecc407cd7" exitCode=0 Jul 13 22:31:02.032169 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:02.032096 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" Jul 13 22:31:02.032169 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:02.032122 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" event={"ID":"405aa82e-1358-4ffc-8653-eccb3956c435","Type":"ContainerDied","Data":"541fd371e256da5fa2a04b47cdf00d6f69a0984b028e5dd6dcc1433ecc407cd7"} Jul 13 22:31:02.032169 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:02.032159 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx" event={"ID":"405aa82e-1358-4ffc-8653-eccb3956c435","Type":"ContainerDied","Data":"aec46df46f6189e4b61b1b3e542622a6a3365c20614a0f216b2a17150a2c9664"} Jul 13 22:31:02.032282 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:02.032181 2569 scope.go:117] "RemoveContainer" containerID="541fd371e256da5fa2a04b47cdf00d6f69a0984b028e5dd6dcc1433ecc407cd7" Jul 13 22:31:02.041086 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:02.041070 2569 scope.go:117] "RemoveContainer" containerID="db70aeda20cb3350c6eba806173ea5e02704c4d2eac37f85462a8705cf73f030" Jul 13 22:31:02.047457 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:02.047405 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" podStartSLOduration=2.5210847530000002 podStartE2EDuration="3.047390126s" totalImagesPullingTime="526.305373ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:31:02.047276258 +0000 UTC m=+1974.704481032" watchObservedRunningTime="2026-07-13 22:31:02.047390126 +0000 UTC m=+1974.704594895" Jul 13 22:31:02.049281 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:02.049262 2569 scope.go:117] "RemoveContainer" containerID="541fd371e256da5fa2a04b47cdf00d6f69a0984b028e5dd6dcc1433ecc407cd7" Jul 13 22:31:02.049550 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:31:02.049513 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"541fd371e256da5fa2a04b47cdf00d6f69a0984b028e5dd6dcc1433ecc407cd7\": container with ID starting with 541fd371e256da5fa2a04b47cdf00d6f69a0984b028e5dd6dcc1433ecc407cd7 not found: ID does not exist" containerID="541fd371e256da5fa2a04b47cdf00d6f69a0984b028e5dd6dcc1433ecc407cd7" Jul 13 22:31:02.049687 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:02.049663 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541fd371e256da5fa2a04b47cdf00d6f69a0984b028e5dd6dcc1433ecc407cd7"} err="failed to get container status \"541fd371e256da5fa2a04b47cdf00d6f69a0984b028e5dd6dcc1433ecc407cd7\": rpc error: code = NotFound desc = could not find container \"541fd371e256da5fa2a04b47cdf00d6f69a0984b028e5dd6dcc1433ecc407cd7\": container with ID starting with 541fd371e256da5fa2a04b47cdf00d6f69a0984b028e5dd6dcc1433ecc407cd7 not found: ID does not exist" Jul 13 22:31:02.049767 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:02.049691 2569 scope.go:117] "RemoveContainer" containerID="db70aeda20cb3350c6eba806173ea5e02704c4d2eac37f85462a8705cf73f030" Jul 13 22:31:02.049942 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:31:02.049920 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db70aeda20cb3350c6eba806173ea5e02704c4d2eac37f85462a8705cf73f030\": container with ID starting with db70aeda20cb3350c6eba806173ea5e02704c4d2eac37f85462a8705cf73f030 not found: ID does not exist" containerID="db70aeda20cb3350c6eba806173ea5e02704c4d2eac37f85462a8705cf73f030" Jul 13 22:31:02.049990 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:02.049949 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db70aeda20cb3350c6eba806173ea5e02704c4d2eac37f85462a8705cf73f030"} err="failed to get container status \"db70aeda20cb3350c6eba806173ea5e02704c4d2eac37f85462a8705cf73f030\": rpc error: code = NotFound desc = could not find container \"db70aeda20cb3350c6eba806173ea5e02704c4d2eac37f85462a8705cf73f030\": container with ID starting with db70aeda20cb3350c6eba806173ea5e02704c4d2eac37f85462a8705cf73f030 not found: ID does not exist" Jul 13 22:31:02.060970 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:02.060949 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx"] Jul 13 22:31:02.065651 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:02.065630 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-61b7b8-openshift-default-5bbb59f59f-qprdx"] Jul 13 22:31:03.931387 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:03.931349 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="405aa82e-1358-4ffc-8653-eccb3956c435" path="/var/lib/kubelet/pods/405aa82e-1358-4ffc-8653-eccb3956c435/volumes" Jul 13 22:31:05.320512 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.320482 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w"] Jul 13 22:31:05.321029 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.320905 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="405aa82e-1358-4ffc-8653-eccb3956c435" containerName="istio-proxy" Jul 13 22:31:05.321029 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.320918 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="405aa82e-1358-4ffc-8653-eccb3956c435" containerName="istio-proxy" Jul 13 22:31:05.321029 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.320994 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="405aa82e-1358-4ffc-8653-eccb3956c435" containerName="istio-proxy" Jul 13 22:31:05.325344 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.325325 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.328507 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.328485 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d07445-openshift-default-dockercfg-mkpwt\"" Jul 13 22:31:05.343822 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.343796 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w"] Jul 13 22:31:05.429556 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.429506 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-token\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.429726 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.429618 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcxvf\" (UniqueName: \"kubernetes.io/projected/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-kube-api-access-xcxvf\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.429726 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.429678 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istiod-ca-cert\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.429811 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.429740 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-credential-socket\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.429851 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.429834 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-data\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.429901 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.429870 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-workload-socket\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.429950 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.429899 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-workload-certs\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.429950 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.429922 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-podinfo\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.430029 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.429974 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-envoy\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.530617 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.530583 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-data\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.530617 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.530622 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-workload-socket\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.530826 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.530641 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-workload-certs\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.530826 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.530671 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-podinfo\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.530826 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.530711 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-envoy\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.530826 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.530741 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-token\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.530826 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.530773 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcxvf\" (UniqueName: \"kubernetes.io/projected/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-kube-api-access-xcxvf\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.530826 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.530806 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istiod-ca-cert\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.531124 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.530849 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-credential-socket\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.531124 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.531022 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-data\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.531232 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.531105 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-workload-socket\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.531232 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.531193 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-credential-socket\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.531344 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.531306 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-workload-certs\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.531716 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.531691 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istiod-ca-cert\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.532909 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.532890 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-envoy\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.533297 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.533277 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-podinfo\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.557593 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.557567 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-token\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.559801 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.559777 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcxvf\" (UniqueName: \"kubernetes.io/projected/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-kube-api-access-xcxvf\") pod \"e2e-api-b-d07445-openshift-default-6874b7d469-tp79w\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.635962 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.635903 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:05.993192 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.993167 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w"] Jul 13 22:31:05.995040 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:31:05.995013 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac6b5da_7d4b_4617_a8ed_7f818b8aa4b2.slice/crio-e0facbfe60b74a11492f0c6bfce402b4db6ac9f772203d327e6ec691b8d42046 WatchSource:0}: Error finding container e0facbfe60b74a11492f0c6bfce402b4db6ac9f772203d327e6ec691b8d42046: Status 404 returned error can't find the container with id e0facbfe60b74a11492f0c6bfce402b4db6ac9f772203d327e6ec691b8d42046 Jul 13 22:31:05.997105 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.997064 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:31:05.997199 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.997131 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:31:05.997199 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:05.997159 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:31:06.053487 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:06.053462 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" event={"ID":"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2","Type":"ContainerStarted","Data":"e0facbfe60b74a11492f0c6bfce402b4db6ac9f772203d327e6ec691b8d42046"} Jul 13 22:31:07.059946 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:07.059901 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" event={"ID":"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2","Type":"ContainerStarted","Data":"1ec93cd55f546e0ae4f00189c2219741f2a6d53b882aa4eab22142f5831588ff"} Jul 13 22:31:07.086262 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:07.086205 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" podStartSLOduration=2.086184836 podStartE2EDuration="2.086184836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:31:07.083804904 +0000 UTC m=+1979.741009673" watchObservedRunningTime="2026-07-13 22:31:07.086184836 +0000 UTC m=+1979.743389604" Jul 13 22:31:07.636558 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:07.636504 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:07.641691 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:07.641663 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:08.043433 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:08.043408 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" Jul 13 22:31:08.065744 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:08.065716 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:08.066622 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:08.066601 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:09.320390 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:09.320359 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm"] Jul 13 22:31:09.320843 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:09.320829 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="405aa82e-1358-4ffc-8653-eccb3956c435" containerName="istio-proxy" Jul 13 22:31:09.320901 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:09.320845 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="405aa82e-1358-4ffc-8653-eccb3956c435" containerName="istio-proxy" Jul 13 22:31:09.320939 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:09.320925 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="405aa82e-1358-4ffc-8653-eccb3956c435" containerName="istio-proxy" Jul 13 22:31:09.325315 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:09.325295 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" Jul 13 22:31:09.328305 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:09.328278 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d07445-serving-cert\"" Jul 13 22:31:09.341225 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:09.341202 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm"] Jul 13 22:31:09.366839 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:09.366650 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d88058-f7de-4ce1-907a-dd116e8b3880-maas-api-tls\") pod \"maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm\" (UID: \"b7d88058-f7de-4ce1-907a-dd116e8b3880\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" Jul 13 22:31:09.366839 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:09.366708 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d88058-f7de-4ce1-907a-dd116e8b3880-openshift-service-ca\") pod \"maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm\" (UID: \"b7d88058-f7de-4ce1-907a-dd116e8b3880\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" Jul 13 22:31:09.366839 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:09.366823 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27wwp\" (UniqueName: \"kubernetes.io/projected/b7d88058-f7de-4ce1-907a-dd116e8b3880-kube-api-access-27wwp\") pod \"maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm\" (UID: \"b7d88058-f7de-4ce1-907a-dd116e8b3880\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" Jul 13 22:31:09.467834 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:09.467801 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d88058-f7de-4ce1-907a-dd116e8b3880-maas-api-tls\") pod \"maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm\" (UID: \"b7d88058-f7de-4ce1-907a-dd116e8b3880\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" Jul 13 22:31:09.468017 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:09.467851 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d88058-f7de-4ce1-907a-dd116e8b3880-openshift-service-ca\") pod \"maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm\" (UID: \"b7d88058-f7de-4ce1-907a-dd116e8b3880\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" Jul 13 22:31:09.468017 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:09.467919 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27wwp\" (UniqueName: \"kubernetes.io/projected/b7d88058-f7de-4ce1-907a-dd116e8b3880-kube-api-access-27wwp\") pod \"maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm\" (UID: \"b7d88058-f7de-4ce1-907a-dd116e8b3880\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" Jul 13 22:31:09.468017 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:31:09.467956 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-serving-cert: secret "maas-api-e2e-api-b-d07445-serving-cert" not found Jul 13 22:31:09.468173 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:31:09.468027 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d88058-f7de-4ce1-907a-dd116e8b3880-maas-api-tls podName:b7d88058-f7de-4ce1-907a-dd116e8b3880 nodeName:}" failed. No retries permitted until 2026-07-13 22:31:09.968006257 +0000 UTC m=+1982.625211003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7d88058-f7de-4ce1-907a-dd116e8b3880-maas-api-tls") pod "maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" (UID: "b7d88058-f7de-4ce1-907a-dd116e8b3880") : secret "maas-api-e2e-api-b-d07445-serving-cert" not found Jul 13 22:31:09.468630 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:09.468613 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d88058-f7de-4ce1-907a-dd116e8b3880-openshift-service-ca\") pod \"maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm\" (UID: \"b7d88058-f7de-4ce1-907a-dd116e8b3880\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" Jul 13 22:31:09.496115 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:09.496081 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27wwp\" (UniqueName: \"kubernetes.io/projected/b7d88058-f7de-4ce1-907a-dd116e8b3880-kube-api-access-27wwp\") pod \"maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm\" (UID: \"b7d88058-f7de-4ce1-907a-dd116e8b3880\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" Jul 13 22:31:09.972736 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:09.972700 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d88058-f7de-4ce1-907a-dd116e8b3880-maas-api-tls\") pod \"maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm\" (UID: \"b7d88058-f7de-4ce1-907a-dd116e8b3880\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" Jul 13 22:31:09.975280 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:09.975250 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d88058-f7de-4ce1-907a-dd116e8b3880-maas-api-tls\") pod \"maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm\" (UID: \"b7d88058-f7de-4ce1-907a-dd116e8b3880\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" Jul 13 22:31:10.236123 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:10.236019 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" Jul 13 22:31:10.431560 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:10.431515 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm"] Jul 13 22:31:10.433573 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:31:10.433522 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d88058_f7de_4ce1_907a_dd116e8b3880.slice/crio-7172271ce512bfc5fcd51d14b1e363736dacf522d0b625fac324a0b02acae520 WatchSource:0}: Error finding container 7172271ce512bfc5fcd51d14b1e363736dacf522d0b625fac324a0b02acae520: Status 404 returned error can't find the container with id 7172271ce512bfc5fcd51d14b1e363736dacf522d0b625fac324a0b02acae520 Jul 13 22:31:11.087797 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:11.087754 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" event={"ID":"b7d88058-f7de-4ce1-907a-dd116e8b3880","Type":"ContainerStarted","Data":"19872780f487ba86c506dd6ac861e5edc4eab0d49942ac0818ca5a9e6457a05c"} Jul 13 22:31:11.087996 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:11.087860 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" event={"ID":"b7d88058-f7de-4ce1-907a-dd116e8b3880","Type":"ContainerStarted","Data":"7172271ce512bfc5fcd51d14b1e363736dacf522d0b625fac324a0b02acae520"} Jul 13 22:31:11.087996 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:11.087932 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" Jul 13 22:31:11.121768 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:11.121718 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" podStartSLOduration=1.6629425 podStartE2EDuration="2.121702928s" totalImagesPullingTime="458.760428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:31:11.119818463 +0000 UTC m=+1983.777023291" watchObservedRunningTime="2026-07-13 22:31:11.121702928 +0000 UTC m=+1983.778907696" Jul 13 22:31:15.049754 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:15.049710 2569 patch_prober.go:28] interesting pod/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 13 22:31:15.050168 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:15.049777 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" podUID="27058e4c-0e92-4347-96d7-3ffa6a52ac9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 13 22:31:17.101789 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:17.101764 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" Jul 13 22:31:17.134350 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:17.134316 2569 generic.go:356] "Generic (PLEG): container finished" podID="27058e4c-0e92-4347-96d7-3ffa6a52ac9e" containerID="aa103942da049c91ef50360405e3bc074545605308bb8ff80e1152425765e077" exitCode=0 Jul 13 22:31:17.134517 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:17.134392 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" event={"ID":"27058e4c-0e92-4347-96d7-3ffa6a52ac9e","Type":"ContainerDied","Data":"aa103942da049c91ef50360405e3bc074545605308bb8ff80e1152425765e077"} Jul 13 22:31:17.134926 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:17.134907 2569 scope.go:117] "RemoveContainer" containerID="aa103942da049c91ef50360405e3bc074545605308bb8ff80e1152425765e077" Jul 13 22:31:17.136297 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:17.136263 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:31:17.136418 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:17.136338 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:31:17.136418 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:17.136385 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:31:18.144766 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:18.144726 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" event={"ID":"27058e4c-0e92-4347-96d7-3ffa6a52ac9e","Type":"ContainerStarted","Data":"df19985e0a117cd1cbf4ae70a27aa154e25d852aa2e5a2c255ddff7f6a7c815f"} Jul 13 22:31:18.657281 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:18.657245 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:31:18.657445 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:18.657414 2569 patch_prober.go:28] interesting pod/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 13 22:31:18.657512 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:18.657475 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" podUID="27058e4c-0e92-4347-96d7-3ffa6a52ac9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 13 22:31:19.656902 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:19.656867 2569 patch_prober.go:28] interesting pod/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 13 22:31:19.657296 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:19.656919 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" podUID="27058e4c-0e92-4347-96d7-3ffa6a52ac9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 13 22:31:20.657146 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:20.657077 2569 patch_prober.go:28] interesting pod/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 13 22:31:20.657146 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:20.657139 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" podUID="27058e4c-0e92-4347-96d7-3ffa6a52ac9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 13 22:31:20.817609 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:20.814416 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61b7b8-29733030-57z4c" Jul 13 22:31:21.162371 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:21.162278 2569 generic.go:356] "Generic (PLEG): container finished" podID="27c7db64-1dbe-4d45-87b5-51bb850bf864" containerID="869a16e033fdf485867556d92307e622b7bd58125522314bf04ec074a6854fae" exitCode=137 Jul 13 22:31:21.162371 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:21.162329 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61b7b8-29733030-57z4c" Jul 13 22:31:21.162600 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:21.162361 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61b7b8-29733030-57z4c" event={"ID":"27c7db64-1dbe-4d45-87b5-51bb850bf864","Type":"ContainerDied","Data":"869a16e033fdf485867556d92307e622b7bd58125522314bf04ec074a6854fae"} Jul 13 22:31:21.162600 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:21.162416 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61b7b8-29733030-57z4c" event={"ID":"27c7db64-1dbe-4d45-87b5-51bb850bf864","Type":"ContainerDied","Data":"9c5202ef6527f53f8533a816839c7f13a3538d305b950c4dd244d56e1c5a24f9"} Jul 13 22:31:21.162600 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:21.162438 2569 scope.go:117] "RemoveContainer" containerID="869a16e033fdf485867556d92307e622b7bd58125522314bf04ec074a6854fae" Jul 13 22:31:21.171556 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:21.171524 2569 scope.go:117] "RemoveContainer" containerID="869a16e033fdf485867556d92307e622b7bd58125522314bf04ec074a6854fae" Jul 13 22:31:21.171807 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:31:21.171792 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"869a16e033fdf485867556d92307e622b7bd58125522314bf04ec074a6854fae\": container with ID starting with 869a16e033fdf485867556d92307e622b7bd58125522314bf04ec074a6854fae not found: ID does not exist" containerID="869a16e033fdf485867556d92307e622b7bd58125522314bf04ec074a6854fae" Jul 13 22:31:21.171861 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:21.171814 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869a16e033fdf485867556d92307e622b7bd58125522314bf04ec074a6854fae"} err="failed to get container status \"869a16e033fdf485867556d92307e622b7bd58125522314bf04ec074a6854fae\": rpc error: code = NotFound desc = could not find container \"869a16e033fdf485867556d92307e622b7bd58125522314bf04ec074a6854fae\": container with ID starting with 869a16e033fdf485867556d92307e622b7bd58125522314bf04ec074a6854fae not found: ID does not exist" Jul 13 22:31:21.191663 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:21.191629 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61b7b8-29733030-57z4c"] Jul 13 22:31:21.195474 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:21.195441 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61b7b8-29733030-57z4c"] Jul 13 22:31:21.657360 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:21.657316 2569 patch_prober.go:28] interesting pod/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 13 22:31:21.657807 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:21.657383 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" podUID="27058e4c-0e92-4347-96d7-3ffa6a52ac9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 13 22:31:21.932781 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:21.932702 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27c7db64-1dbe-4d45-87b5-51bb850bf864" path="/var/lib/kubelet/pods/27c7db64-1dbe-4d45-87b5-51bb850bf864/volumes" Jul 13 22:31:22.118477 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:22.118437 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm"] Jul 13 22:31:22.118805 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:22.118760 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" podUID="9f59f22b-774f-4d69-85fe-65494b6bb12b" containerName="maas-api" containerID="cri-o://5bfea4fe8b6e17e3a14923e25abf8e4705220d57d4386ba2cedf38b0de82f452" gracePeriod=30 Jul 13 22:31:22.392210 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:22.392183 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" Jul 13 22:31:22.532905 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:22.532813 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f59f22b-774f-4d69-85fe-65494b6bb12b-openshift-service-ca\") pod \"9f59f22b-774f-4d69-85fe-65494b6bb12b\" (UID: \"9f59f22b-774f-4d69-85fe-65494b6bb12b\") " Jul 13 22:31:22.533069 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:22.532922 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm8sx\" (UniqueName: \"kubernetes.io/projected/9f59f22b-774f-4d69-85fe-65494b6bb12b-kube-api-access-wm8sx\") pod \"9f59f22b-774f-4d69-85fe-65494b6bb12b\" (UID: \"9f59f22b-774f-4d69-85fe-65494b6bb12b\") " Jul 13 22:31:22.533069 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:22.532962 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f59f22b-774f-4d69-85fe-65494b6bb12b-maas-api-tls\") pod \"9f59f22b-774f-4d69-85fe-65494b6bb12b\" (UID: \"9f59f22b-774f-4d69-85fe-65494b6bb12b\") " Jul 13 22:31:22.533257 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:22.533230 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f59f22b-774f-4d69-85fe-65494b6bb12b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f59f22b-774f-4d69-85fe-65494b6bb12b" (UID: "9f59f22b-774f-4d69-85fe-65494b6bb12b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:31:22.533360 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:22.533348 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f59f22b-774f-4d69-85fe-65494b6bb12b-openshift-service-ca\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:22.535051 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:22.535030 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f59f22b-774f-4d69-85fe-65494b6bb12b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f59f22b-774f-4d69-85fe-65494b6bb12b" (UID: "9f59f22b-774f-4d69-85fe-65494b6bb12b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:31:22.535311 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:22.535296 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f59f22b-774f-4d69-85fe-65494b6bb12b-kube-api-access-wm8sx" (OuterVolumeSpecName: "kube-api-access-wm8sx") pod "9f59f22b-774f-4d69-85fe-65494b6bb12b" (UID: "9f59f22b-774f-4d69-85fe-65494b6bb12b"). InnerVolumeSpecName "kube-api-access-wm8sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:31:22.634823 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:22.634786 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm8sx\" (UniqueName: \"kubernetes.io/projected/9f59f22b-774f-4d69-85fe-65494b6bb12b-kube-api-access-wm8sx\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:22.634823 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:22.634819 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f59f22b-774f-4d69-85fe-65494b6bb12b-maas-api-tls\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:22.661360 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:22.661324 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:31:22.661852 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:22.661654 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:31:22.662325 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:22.662309 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:31:23.175283 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:23.175244 2569 generic.go:356] "Generic (PLEG): container finished" podID="9f59f22b-774f-4d69-85fe-65494b6bb12b" containerID="5bfea4fe8b6e17e3a14923e25abf8e4705220d57d4386ba2cedf38b0de82f452" exitCode=0 Jul 13 22:31:23.175469 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:23.175319 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" event={"ID":"9f59f22b-774f-4d69-85fe-65494b6bb12b","Type":"ContainerDied","Data":"5bfea4fe8b6e17e3a14923e25abf8e4705220d57d4386ba2cedf38b0de82f452"} Jul 13 22:31:23.175469 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:23.175324 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" Jul 13 22:31:23.175469 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:23.175405 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm" event={"ID":"9f59f22b-774f-4d69-85fe-65494b6bb12b","Type":"ContainerDied","Data":"1c8d790758f11b53cb748c184617b1de98f9180fb234fc2094360f7b01655fa9"} Jul 13 22:31:23.175469 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:23.175429 2569 scope.go:117] "RemoveContainer" containerID="5bfea4fe8b6e17e3a14923e25abf8e4705220d57d4386ba2cedf38b0de82f452" Jul 13 22:31:23.187831 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:23.187808 2569 scope.go:117] "RemoveContainer" containerID="5bfea4fe8b6e17e3a14923e25abf8e4705220d57d4386ba2cedf38b0de82f452" Jul 13 22:31:23.188211 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:31:23.188187 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bfea4fe8b6e17e3a14923e25abf8e4705220d57d4386ba2cedf38b0de82f452\": container with ID starting with 5bfea4fe8b6e17e3a14923e25abf8e4705220d57d4386ba2cedf38b0de82f452 not found: ID does not exist" containerID="5bfea4fe8b6e17e3a14923e25abf8e4705220d57d4386ba2cedf38b0de82f452" Jul 13 22:31:23.188299 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:23.188220 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bfea4fe8b6e17e3a14923e25abf8e4705220d57d4386ba2cedf38b0de82f452"} err="failed to get container status \"5bfea4fe8b6e17e3a14923e25abf8e4705220d57d4386ba2cedf38b0de82f452\": rpc error: code = NotFound desc = could not find container \"5bfea4fe8b6e17e3a14923e25abf8e4705220d57d4386ba2cedf38b0de82f452\": container with ID starting with 5bfea4fe8b6e17e3a14923e25abf8e4705220d57d4386ba2cedf38b0de82f452 not found: ID does not exist" Jul 13 22:31:23.203245 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:23.203221 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm"] Jul 13 22:31:23.208493 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:23.208464 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-232fa0-79985b7b79-z4hpm"] Jul 13 22:31:23.933406 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:23.933367 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f59f22b-774f-4d69-85fe-65494b6bb12b" path="/var/lib/kubelet/pods/9f59f22b-774f-4d69-85fe-65494b6bb12b/volumes" Jul 13 22:31:24.107692 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:24.107654 2569 patch_prober.go:28] interesting pod/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 13 22:31:24.107864 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:24.107733 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" podUID="7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 13 22:31:25.190832 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:25.190799 2569 generic.go:356] "Generic (PLEG): container finished" podID="7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" containerID="1ec93cd55f546e0ae4f00189c2219741f2a6d53b882aa4eab22142f5831588ff" exitCode=0 Jul 13 22:31:25.191252 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:25.190859 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" event={"ID":"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2","Type":"ContainerDied","Data":"1ec93cd55f546e0ae4f00189c2219741f2a6d53b882aa4eab22142f5831588ff"} Jul 13 22:31:25.191311 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:25.191262 2569 scope.go:117] "RemoveContainer" containerID="1ec93cd55f546e0ae4f00189c2219741f2a6d53b882aa4eab22142f5831588ff" Jul 13 22:31:25.192270 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:25.192242 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:31:25.192330 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:25.192309 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:31:25.192365 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:25.192345 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:31:26.201043 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:26.201002 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" event={"ID":"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2","Type":"ContainerStarted","Data":"54a4a34539acb37f5656a67009e8f53bb6f01a4b81bee56478642d44ab44123d"} Jul 13 22:31:26.636847 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:26.636755 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:26.637053 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:26.637030 2569 patch_prober.go:28] interesting pod/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 13 22:31:26.637100 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:26.637077 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" podUID="7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 13 22:31:27.636667 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:27.636635 2569 patch_prober.go:28] interesting pod/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 13 22:31:27.637048 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:27.636689 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" podUID="7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 13 22:31:28.636798 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:28.636763 2569 patch_prober.go:28] interesting pod/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 13 22:31:28.637195 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:28.636822 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" podUID="7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 13 22:31:28.922858 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:28.922767 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4"] Jul 13 22:31:28.923071 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:28.923038 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" podUID="27058e4c-0e92-4347-96d7-3ffa6a52ac9e" containerName="istio-proxy" containerID="cri-o://df19985e0a117cd1cbf4ae70a27aa154e25d852aa2e5a2c255ddff7f6a7c815f" gracePeriod=30 Jul 13 22:31:29.636604 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:29.636567 2569 patch_prober.go:28] interesting pod/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 13 22:31:29.636779 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:29.636625 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" podUID="7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 13 22:31:30.220020 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.219995 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733030-jql45" Jul 13 22:31:30.223572 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.223512 2569 generic.go:356] "Generic (PLEG): container finished" podID="d885b15c-833a-4905-98c5-df9c702ed461" containerID="44f485d016e5f6f8351b66b3bb8b589d9977a4a9b9eba89d71f6dc5f58bf295e" exitCode=137 Jul 13 22:31:30.223685 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.223570 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733030-jql45" Jul 13 22:31:30.223685 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.223572 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733030-jql45" event={"ID":"d885b15c-833a-4905-98c5-df9c702ed461","Type":"ContainerDied","Data":"44f485d016e5f6f8351b66b3bb8b589d9977a4a9b9eba89d71f6dc5f58bf295e"} Jul 13 22:31:30.223685 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.223623 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733030-jql45" event={"ID":"d885b15c-833a-4905-98c5-df9c702ed461","Type":"ContainerDied","Data":"5a1269e9c2e242cbb38e1acf6767747ed1d5be4c8aa6ebdf06401619b0c898d5"} Jul 13 22:31:30.223685 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.223648 2569 scope.go:117] "RemoveContainer" containerID="44f485d016e5f6f8351b66b3bb8b589d9977a4a9b9eba89d71f6dc5f58bf295e" Jul 13 22:31:30.234851 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.234826 2569 scope.go:117] "RemoveContainer" containerID="44f485d016e5f6f8351b66b3bb8b589d9977a4a9b9eba89d71f6dc5f58bf295e" Jul 13 22:31:30.235305 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:31:30.235282 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f485d016e5f6f8351b66b3bb8b589d9977a4a9b9eba89d71f6dc5f58bf295e\": container with ID starting with 44f485d016e5f6f8351b66b3bb8b589d9977a4a9b9eba89d71f6dc5f58bf295e not found: ID does not exist" containerID="44f485d016e5f6f8351b66b3bb8b589d9977a4a9b9eba89d71f6dc5f58bf295e" Jul 13 22:31:30.235392 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.235316 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f485d016e5f6f8351b66b3bb8b589d9977a4a9b9eba89d71f6dc5f58bf295e"} err="failed to get container status \"44f485d016e5f6f8351b66b3bb8b589d9977a4a9b9eba89d71f6dc5f58bf295e\": rpc error: code = NotFound desc = could not find container \"44f485d016e5f6f8351b66b3bb8b589d9977a4a9b9eba89d71f6dc5f58bf295e\": container with ID starting with 44f485d016e5f6f8351b66b3bb8b589d9977a4a9b9eba89d71f6dc5f58bf295e not found: ID does not exist" Jul 13 22:31:30.287564 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.287445 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733030-jql45"] Jul 13 22:31:30.289429 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.289395 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733030-jql45"] Jul 13 22:31:30.292516 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.292494 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm"] Jul 13 22:31:30.292728 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.292702 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" podUID="b7d88058-f7de-4ce1-907a-dd116e8b3880" containerName="maas-api" containerID="cri-o://19872780f487ba86c506dd6ac861e5edc4eab0d49942ac0818ca5a9e6457a05c" gracePeriod=30 Jul 13 22:31:30.539148 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.539092 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" Jul 13 22:31:30.640250 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.640217 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:30.640565 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.640525 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:30.641093 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.641077 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:30.720832 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.720802 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d88058-f7de-4ce1-907a-dd116e8b3880-openshift-service-ca\") pod \"b7d88058-f7de-4ce1-907a-dd116e8b3880\" (UID: \"b7d88058-f7de-4ce1-907a-dd116e8b3880\") " Jul 13 22:31:30.720832 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.720840 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d88058-f7de-4ce1-907a-dd116e8b3880-maas-api-tls\") pod \"b7d88058-f7de-4ce1-907a-dd116e8b3880\" (UID: \"b7d88058-f7de-4ce1-907a-dd116e8b3880\") " Jul 13 22:31:30.721094 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.720947 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27wwp\" (UniqueName: \"kubernetes.io/projected/b7d88058-f7de-4ce1-907a-dd116e8b3880-kube-api-access-27wwp\") pod \"b7d88058-f7de-4ce1-907a-dd116e8b3880\" (UID: \"b7d88058-f7de-4ce1-907a-dd116e8b3880\") " Jul 13 22:31:30.721164 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.721144 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d88058-f7de-4ce1-907a-dd116e8b3880-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b7d88058-f7de-4ce1-907a-dd116e8b3880" (UID: "b7d88058-f7de-4ce1-907a-dd116e8b3880"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:31:30.723091 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.723062 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d88058-f7de-4ce1-907a-dd116e8b3880-kube-api-access-27wwp" (OuterVolumeSpecName: "kube-api-access-27wwp") pod "b7d88058-f7de-4ce1-907a-dd116e8b3880" (UID: "b7d88058-f7de-4ce1-907a-dd116e8b3880"). InnerVolumeSpecName "kube-api-access-27wwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:31:30.723091 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.723067 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d88058-f7de-4ce1-907a-dd116e8b3880-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b7d88058-f7de-4ce1-907a-dd116e8b3880" (UID: "b7d88058-f7de-4ce1-907a-dd116e8b3880"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:31:30.822203 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.822106 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d88058-f7de-4ce1-907a-dd116e8b3880-openshift-service-ca\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:30.822203 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.822131 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d88058-f7de-4ce1-907a-dd116e8b3880-maas-api-tls\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:30.822203 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:30.822141 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27wwp\" (UniqueName: \"kubernetes.io/projected/b7d88058-f7de-4ce1-907a-dd116e8b3880-kube-api-access-27wwp\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:31.230652 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:31.230615 2569 generic.go:356] "Generic (PLEG): container finished" podID="b7d88058-f7de-4ce1-907a-dd116e8b3880" containerID="19872780f487ba86c506dd6ac861e5edc4eab0d49942ac0818ca5a9e6457a05c" exitCode=0 Jul 13 22:31:31.231142 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:31.230662 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" event={"ID":"b7d88058-f7de-4ce1-907a-dd116e8b3880","Type":"ContainerDied","Data":"19872780f487ba86c506dd6ac861e5edc4eab0d49942ac0818ca5a9e6457a05c"} Jul 13 22:31:31.231142 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:31.230707 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" Jul 13 22:31:31.231142 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:31.230735 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm" event={"ID":"b7d88058-f7de-4ce1-907a-dd116e8b3880","Type":"ContainerDied","Data":"7172271ce512bfc5fcd51d14b1e363736dacf522d0b625fac324a0b02acae520"} Jul 13 22:31:31.231142 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:31.230766 2569 scope.go:117] "RemoveContainer" containerID="19872780f487ba86c506dd6ac861e5edc4eab0d49942ac0818ca5a9e6457a05c" Jul 13 22:31:31.243081 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:31.243055 2569 scope.go:117] "RemoveContainer" containerID="19872780f487ba86c506dd6ac861e5edc4eab0d49942ac0818ca5a9e6457a05c" Jul 13 22:31:31.243349 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:31:31.243328 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19872780f487ba86c506dd6ac861e5edc4eab0d49942ac0818ca5a9e6457a05c\": container with ID starting with 19872780f487ba86c506dd6ac861e5edc4eab0d49942ac0818ca5a9e6457a05c not found: ID does not exist" containerID="19872780f487ba86c506dd6ac861e5edc4eab0d49942ac0818ca5a9e6457a05c" Jul 13 22:31:31.243425 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:31.243357 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19872780f487ba86c506dd6ac861e5edc4eab0d49942ac0818ca5a9e6457a05c"} err="failed to get container status \"19872780f487ba86c506dd6ac861e5edc4eab0d49942ac0818ca5a9e6457a05c\": rpc error: code = NotFound desc = could not find container \"19872780f487ba86c506dd6ac861e5edc4eab0d49942ac0818ca5a9e6457a05c\": container with ID starting with 19872780f487ba86c506dd6ac861e5edc4eab0d49942ac0818ca5a9e6457a05c not found: ID does not exist" Jul 13 22:31:31.255526 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:31.255496 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm"] Jul 13 22:31:31.263098 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:31:31.260254 2569 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d88058_f7de_4ce1_907a_dd116e8b3880.slice\": RecentStats: unable to find data in memory cache]" Jul 13 22:31:31.263098 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:31:31.260375 2569 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d88058_f7de_4ce1_907a_dd116e8b3880.slice\": RecentStats: unable to find data in memory cache]" Jul 13 22:31:31.270927 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:31.270901 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d07445-7df9d997fb-fcsfm"] Jul 13 22:31:31.931629 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:31.931594 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7d88058-f7de-4ce1-907a-dd116e8b3880" path="/var/lib/kubelet/pods/b7d88058-f7de-4ce1-907a-dd116e8b3880/volumes" Jul 13 22:31:31.931878 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:31.931864 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d885b15c-833a-4905-98c5-df9c702ed461" path="/var/lib/kubelet/pods/d885b15c-833a-4905-98c5-df9c702ed461/volumes" Jul 13 22:31:34.189585 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.189496 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:31:34.249102 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.249064 2569 generic.go:356] "Generic (PLEG): container finished" podID="27058e4c-0e92-4347-96d7-3ffa6a52ac9e" containerID="df19985e0a117cd1cbf4ae70a27aa154e25d852aa2e5a2c255ddff7f6a7c815f" exitCode=0 Jul 13 22:31:34.249271 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.249118 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" Jul 13 22:31:34.249271 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.249155 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" event={"ID":"27058e4c-0e92-4347-96d7-3ffa6a52ac9e","Type":"ContainerDied","Data":"df19985e0a117cd1cbf4ae70a27aa154e25d852aa2e5a2c255ddff7f6a7c815f"} Jul 13 22:31:34.249271 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.249180 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istiod-ca-cert\") pod \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " Jul 13 22:31:34.249271 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.249208 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4" event={"ID":"27058e4c-0e92-4347-96d7-3ffa6a52ac9e","Type":"ContainerDied","Data":"00dc692a8e86ebcbc1eeaee48efa8514bcc89ea194672aa4b16c69902e90e9fb"} Jul 13 22:31:34.249271 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.249236 2569 scope.go:117] "RemoveContainer" containerID="df19985e0a117cd1cbf4ae70a27aa154e25d852aa2e5a2c255ddff7f6a7c815f" Jul 13 22:31:34.249271 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.249243 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-data\") pod \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " Jul 13 22:31:34.249644 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.249284 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-podinfo\") pod \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " Jul 13 22:31:34.249644 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.249309 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-envoy\") pod \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " Jul 13 22:31:34.249644 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.249345 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kzd9\" (UniqueName: \"kubernetes.io/projected/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-kube-api-access-6kzd9\") pod \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " Jul 13 22:31:34.249644 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.249378 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-token\") pod \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " Jul 13 22:31:34.249644 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.249448 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-workload-socket\") pod \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " Jul 13 22:31:34.249644 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.249477 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-workload-certs\") pod \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " Jul 13 22:31:34.249644 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.249514 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-credential-socket\") pod \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\" (UID: \"27058e4c-0e92-4347-96d7-3ffa6a52ac9e\") " Jul 13 22:31:34.250018 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.249639 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "27058e4c-0e92-4347-96d7-3ffa6a52ac9e" (UID: "27058e4c-0e92-4347-96d7-3ffa6a52ac9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:31:34.250018 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.249893 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istiod-ca-cert\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:34.250018 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.249994 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "27058e4c-0e92-4347-96d7-3ffa6a52ac9e" (UID: "27058e4c-0e92-4347-96d7-3ffa6a52ac9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:31:34.250179 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.250104 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "27058e4c-0e92-4347-96d7-3ffa6a52ac9e" (UID: "27058e4c-0e92-4347-96d7-3ffa6a52ac9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:31:34.250371 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.250343 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "27058e4c-0e92-4347-96d7-3ffa6a52ac9e" (UID: "27058e4c-0e92-4347-96d7-3ffa6a52ac9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:31:34.250691 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.250617 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-data" (OuterVolumeSpecName: "istio-data") pod "27058e4c-0e92-4347-96d7-3ffa6a52ac9e" (UID: "27058e4c-0e92-4347-96d7-3ffa6a52ac9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:31:34.251897 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.251872 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-token" (OuterVolumeSpecName: "istio-token") pod "27058e4c-0e92-4347-96d7-3ffa6a52ac9e" (UID: "27058e4c-0e92-4347-96d7-3ffa6a52ac9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:31:34.252012 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.251919 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-kube-api-access-6kzd9" (OuterVolumeSpecName: "kube-api-access-6kzd9") pod "27058e4c-0e92-4347-96d7-3ffa6a52ac9e" (UID: "27058e4c-0e92-4347-96d7-3ffa6a52ac9e"). InnerVolumeSpecName "kube-api-access-6kzd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:31:34.252103 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.252085 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "27058e4c-0e92-4347-96d7-3ffa6a52ac9e" (UID: "27058e4c-0e92-4347-96d7-3ffa6a52ac9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:31:34.252182 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.252163 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "27058e4c-0e92-4347-96d7-3ffa6a52ac9e" (UID: "27058e4c-0e92-4347-96d7-3ffa6a52ac9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:31:34.272725 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.272702 2569 scope.go:117] "RemoveContainer" containerID="aa103942da049c91ef50360405e3bc074545605308bb8ff80e1152425765e077" Jul 13 22:31:34.281316 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.281293 2569 scope.go:117] "RemoveContainer" containerID="df19985e0a117cd1cbf4ae70a27aa154e25d852aa2e5a2c255ddff7f6a7c815f" Jul 13 22:31:34.281637 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:31:34.281603 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df19985e0a117cd1cbf4ae70a27aa154e25d852aa2e5a2c255ddff7f6a7c815f\": container with ID starting with df19985e0a117cd1cbf4ae70a27aa154e25d852aa2e5a2c255ddff7f6a7c815f not found: ID does not exist" containerID="df19985e0a117cd1cbf4ae70a27aa154e25d852aa2e5a2c255ddff7f6a7c815f" Jul 13 22:31:34.281753 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.281638 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df19985e0a117cd1cbf4ae70a27aa154e25d852aa2e5a2c255ddff7f6a7c815f"} err="failed to get container status \"df19985e0a117cd1cbf4ae70a27aa154e25d852aa2e5a2c255ddff7f6a7c815f\": rpc error: code = NotFound desc = could not find container \"df19985e0a117cd1cbf4ae70a27aa154e25d852aa2e5a2c255ddff7f6a7c815f\": container with ID starting with df19985e0a117cd1cbf4ae70a27aa154e25d852aa2e5a2c255ddff7f6a7c815f not found: ID does not exist" Jul 13 22:31:34.281753 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.281661 2569 scope.go:117] "RemoveContainer" containerID="aa103942da049c91ef50360405e3bc074545605308bb8ff80e1152425765e077" Jul 13 22:31:34.281940 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:31:34.281920 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa103942da049c91ef50360405e3bc074545605308bb8ff80e1152425765e077\": container with ID starting with aa103942da049c91ef50360405e3bc074545605308bb8ff80e1152425765e077 not found: ID does not exist" containerID="aa103942da049c91ef50360405e3bc074545605308bb8ff80e1152425765e077" Jul 13 22:31:34.282002 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.281945 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa103942da049c91ef50360405e3bc074545605308bb8ff80e1152425765e077"} err="failed to get container status \"aa103942da049c91ef50360405e3bc074545605308bb8ff80e1152425765e077\": rpc error: code = NotFound desc = could not find container \"aa103942da049c91ef50360405e3bc074545605308bb8ff80e1152425765e077\": container with ID starting with aa103942da049c91ef50360405e3bc074545605308bb8ff80e1152425765e077 not found: ID does not exist" Jul 13 22:31:34.350406 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.350360 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-workload-certs\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:34.350406 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.350404 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-credential-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:34.350406 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.350415 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-data\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:34.350685 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.350425 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-podinfo\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:34.350685 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.350435 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-envoy\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:34.350685 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.350447 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kzd9\" (UniqueName: \"kubernetes.io/projected/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-kube-api-access-6kzd9\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:34.350685 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.350461 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-istio-token\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:34.350685 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.350474 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27058e4c-0e92-4347-96d7-3ffa6a52ac9e-workload-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:34.571307 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.571279 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4"] Jul 13 22:31:34.576760 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:34.576735 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-232fa0-openshift-default-6b45699f5f-zw9r4"] Jul 13 22:31:35.931088 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:35.931059 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27058e4c-0e92-4347-96d7-3ffa6a52ac9e" path="/var/lib/kubelet/pods/27058e4c-0e92-4347-96d7-3ffa6a52ac9e/volumes" Jul 13 22:31:36.292021 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:36.291993 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w"] Jul 13 22:31:36.292252 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:36.292224 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" podUID="7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" containerName="istio-proxy" containerID="cri-o://54a4a34539acb37f5656a67009e8f53bb6f01a4b81bee56478642d44ab44123d" gracePeriod=30 Jul 13 22:31:37.171747 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.171716 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t"] Jul 13 22:31:37.172131 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.172122 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27058e4c-0e92-4347-96d7-3ffa6a52ac9e" containerName="istio-proxy" Jul 13 22:31:37.172187 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.172133 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="27058e4c-0e92-4347-96d7-3ffa6a52ac9e" containerName="istio-proxy" Jul 13 22:31:37.172187 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.172151 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f59f22b-774f-4d69-85fe-65494b6bb12b" containerName="maas-api" Jul 13 22:31:37.172187 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.172157 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f59f22b-774f-4d69-85fe-65494b6bb12b" containerName="maas-api" Jul 13 22:31:37.172563 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.172516 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d885b15c-833a-4905-98c5-df9c702ed461" containerName="cleanup" Jul 13 22:31:37.172563 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.172553 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d885b15c-833a-4905-98c5-df9c702ed461" containerName="cleanup" Jul 13 22:31:37.172724 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.172599 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27c7db64-1dbe-4d45-87b5-51bb850bf864" containerName="cleanup" Jul 13 22:31:37.172724 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.172609 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="27c7db64-1dbe-4d45-87b5-51bb850bf864" containerName="cleanup" Jul 13 22:31:37.172724 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.172640 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7d88058-f7de-4ce1-907a-dd116e8b3880" containerName="maas-api" Jul 13 22:31:37.172724 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.172649 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7d88058-f7de-4ce1-907a-dd116e8b3880" containerName="maas-api" Jul 13 22:31:37.173015 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.172996 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7d88058-f7de-4ce1-907a-dd116e8b3880" containerName="maas-api" Jul 13 22:31:37.173068 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.173020 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="27058e4c-0e92-4347-96d7-3ffa6a52ac9e" containerName="istio-proxy" Jul 13 22:31:37.173068 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.173029 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="27c7db64-1dbe-4d45-87b5-51bb850bf864" containerName="cleanup" Jul 13 22:31:37.173068 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.173038 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="27058e4c-0e92-4347-96d7-3ffa6a52ac9e" containerName="istio-proxy" Jul 13 22:31:37.173068 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.173061 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f59f22b-774f-4d69-85fe-65494b6bb12b" containerName="maas-api" Jul 13 22:31:37.173250 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.173072 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d885b15c-833a-4905-98c5-df9c702ed461" containerName="cleanup" Jul 13 22:31:37.177008 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.176981 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.182870 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.182845 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-0eaefd-openshift-default-dockercfg-wv9dq\"" Jul 13 22:31:37.191560 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.191512 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t"] Jul 13 22:31:37.274619 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.274518 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzcfk\" (UniqueName: \"kubernetes.io/projected/d5181cf8-453e-4dd6-a378-1a33a198bdce-kube-api-access-jzcfk\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.274619 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.274615 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5181cf8-453e-4dd6-a378-1a33a198bdce-istiod-ca-cert\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.274871 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.274702 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-data\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.274871 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.274746 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-envoy\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.274871 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.274780 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-workload-socket\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.274871 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.274832 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-token\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.274871 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.274858 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-workload-certs\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.275137 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.274909 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-podinfo\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.275137 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.274960 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-credential-socket\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.376062 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.376025 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-podinfo\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.376256 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.376073 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-credential-socket\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.376256 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.376102 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzcfk\" (UniqueName: \"kubernetes.io/projected/d5181cf8-453e-4dd6-a378-1a33a198bdce-kube-api-access-jzcfk\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.376256 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.376129 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5181cf8-453e-4dd6-a378-1a33a198bdce-istiod-ca-cert\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.376256 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.376161 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-data\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.376256 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.376180 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-envoy\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.376522 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.376299 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-workload-socket\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.376522 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.376354 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-token\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.376522 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.376375 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-workload-certs\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.376724 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.376594 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-credential-socket\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.376724 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.376674 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-workload-socket\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.376839 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.376750 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-workload-certs\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.377003 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.376981 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-data\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.377242 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.377219 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5181cf8-453e-4dd6-a378-1a33a198bdce-istiod-ca-cert\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.378722 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.378693 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-envoy\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.380007 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.379978 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-podinfo\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.385122 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.385094 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-token\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.385742 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.385699 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzcfk\" (UniqueName: \"kubernetes.io/projected/d5181cf8-453e-4dd6-a378-1a33a198bdce-kube-api-access-jzcfk\") pod \"e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.494157 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.494135 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:37.628637 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.628610 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t"] Jul 13 22:31:37.630043 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:31:37.630015 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5181cf8_453e_4dd6_a378_1a33a198bdce.slice/crio-49c3d7186c99e7d34978cebfd887ab2c058bd7d7c6b8475bbacaffab56355aad WatchSource:0}: Error finding container 49c3d7186c99e7d34978cebfd887ab2c058bd7d7c6b8475bbacaffab56355aad: Status 404 returned error can't find the container with id 49c3d7186c99e7d34978cebfd887ab2c058bd7d7c6b8475bbacaffab56355aad Jul 13 22:31:37.632169 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.632138 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:31:37.632261 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.632204 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:31:37.632261 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:37.632234 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:31:38.274041 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:38.273999 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" event={"ID":"d5181cf8-453e-4dd6-a378-1a33a198bdce","Type":"ContainerStarted","Data":"7ea6329adad096bdd129752f75e1472685071c6b73eccbcad8ee8e63605c51d3"} Jul 13 22:31:38.274436 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:38.274063 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" event={"ID":"d5181cf8-453e-4dd6-a378-1a33a198bdce","Type":"ContainerStarted","Data":"49c3d7186c99e7d34978cebfd887ab2c058bd7d7c6b8475bbacaffab56355aad"} Jul 13 22:31:38.298439 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:38.298394 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" podStartSLOduration=1.298376838 podStartE2EDuration="1.298376838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:31:38.295752665 +0000 UTC m=+2010.952957434" watchObservedRunningTime="2026-07-13 22:31:38.298376838 +0000 UTC m=+2010.955581677" Jul 13 22:31:38.494412 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:38.494378 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:38.499375 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:38.499350 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:39.280199 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:39.280165 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:39.281294 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:39.281267 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:40.247455 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:40.247418 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf"] Jul 13 22:31:40.247957 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:40.247938 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27058e4c-0e92-4347-96d7-3ffa6a52ac9e" containerName="istio-proxy" Jul 13 22:31:40.247957 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:40.247959 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="27058e4c-0e92-4347-96d7-3ffa6a52ac9e" containerName="istio-proxy" Jul 13 22:31:40.250261 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:40.250241 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" Jul 13 22:31:40.252762 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:40.252744 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-0eaefd-serving-cert\"" Jul 13 22:31:40.262418 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:40.262397 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf"] Jul 13 22:31:40.302841 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:40.302812 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6a4dacf-39a7-4386-9778-c99b08deb5c3-maas-api-tls\") pod \"maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf\" (UID: \"d6a4dacf-39a7-4386-9778-c99b08deb5c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" Jul 13 22:31:40.303184 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:40.302924 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a4dacf-39a7-4386-9778-c99b08deb5c3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf\" (UID: \"d6a4dacf-39a7-4386-9778-c99b08deb5c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" Jul 13 22:31:40.303184 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:40.302993 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4rs4\" (UniqueName: \"kubernetes.io/projected/d6a4dacf-39a7-4386-9778-c99b08deb5c3-kube-api-access-f4rs4\") pod \"maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf\" (UID: \"d6a4dacf-39a7-4386-9778-c99b08deb5c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" Jul 13 22:31:40.403795 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:40.403760 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6a4dacf-39a7-4386-9778-c99b08deb5c3-maas-api-tls\") pod \"maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf\" (UID: \"d6a4dacf-39a7-4386-9778-c99b08deb5c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" Jul 13 22:31:40.403969 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:40.403820 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a4dacf-39a7-4386-9778-c99b08deb5c3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf\" (UID: \"d6a4dacf-39a7-4386-9778-c99b08deb5c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" Jul 13 22:31:40.403969 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:40.403854 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4rs4\" (UniqueName: \"kubernetes.io/projected/d6a4dacf-39a7-4386-9778-c99b08deb5c3-kube-api-access-f4rs4\") pod \"maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf\" (UID: \"d6a4dacf-39a7-4386-9778-c99b08deb5c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" Jul 13 22:31:40.403969 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:31:40.403901 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-serving-cert: secret "maas-api-e2e-shared-a-0eaefd-serving-cert" not found Jul 13 22:31:40.404090 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:31:40.403971 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a4dacf-39a7-4386-9778-c99b08deb5c3-maas-api-tls podName:d6a4dacf-39a7-4386-9778-c99b08deb5c3 nodeName:}" failed. No retries permitted until 2026-07-13 22:31:40.903953847 +0000 UTC m=+2013.561158609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6a4dacf-39a7-4386-9778-c99b08deb5c3-maas-api-tls") pod "maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" (UID: "d6a4dacf-39a7-4386-9778-c99b08deb5c3") : secret "maas-api-e2e-shared-a-0eaefd-serving-cert" not found Jul 13 22:31:40.404614 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:40.404595 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a4dacf-39a7-4386-9778-c99b08deb5c3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf\" (UID: \"d6a4dacf-39a7-4386-9778-c99b08deb5c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" Jul 13 22:31:40.413859 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:40.413837 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4rs4\" (UniqueName: \"kubernetes.io/projected/d6a4dacf-39a7-4386-9778-c99b08deb5c3-kube-api-access-f4rs4\") pod \"maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf\" (UID: \"d6a4dacf-39a7-4386-9778-c99b08deb5c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" Jul 13 22:31:40.906789 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:40.906748 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6a4dacf-39a7-4386-9778-c99b08deb5c3-maas-api-tls\") pod \"maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf\" (UID: \"d6a4dacf-39a7-4386-9778-c99b08deb5c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" Jul 13 22:31:40.909360 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:40.909337 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6a4dacf-39a7-4386-9778-c99b08deb5c3-maas-api-tls\") pod \"maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf\" (UID: \"d6a4dacf-39a7-4386-9778-c99b08deb5c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" Jul 13 22:31:41.162229 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.162146 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" Jul 13 22:31:41.302507 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.302481 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf"] Jul 13 22:31:41.305349 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:31:41.305318 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a4dacf_39a7_4386_9778_c99b08deb5c3.slice/crio-1c69a61b0f353adae0caebadc5926e32de547151eff46d73fc0960705f7dae88 WatchSource:0}: Error finding container 1c69a61b0f353adae0caebadc5926e32de547151eff46d73fc0960705f7dae88: Status 404 returned error can't find the container with id 1c69a61b0f353adae0caebadc5926e32de547151eff46d73fc0960705f7dae88 Jul 13 22:31:41.554062 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.554039 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:41.614288 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.614252 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-token\") pod \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " Jul 13 22:31:41.614488 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.614295 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-workload-socket\") pod \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " Jul 13 22:31:41.614488 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.614331 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcxvf\" (UniqueName: \"kubernetes.io/projected/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-kube-api-access-xcxvf\") pod \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " Jul 13 22:31:41.614488 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.614431 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-envoy\") pod \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " Jul 13 22:31:41.614726 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.614497 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-data\") pod \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " Jul 13 22:31:41.614726 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.614559 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-podinfo\") pod \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " Jul 13 22:31:41.614726 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.614623 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-workload-certs\") pod \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " Jul 13 22:31:41.614726 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.614666 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istiod-ca-cert\") pod \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " Jul 13 22:31:41.614726 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.614664 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" (UID: "7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:31:41.614726 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.614724 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-credential-socket\") pod \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\" (UID: \"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2\") " Jul 13 22:31:41.615090 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.615060 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" (UID: "7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:31:41.615172 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.615102 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-workload-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:41.615309 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.615283 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" (UID: "7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:31:41.615405 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.615382 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" (UID: "7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:31:41.615631 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.615604 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-data" (OuterVolumeSpecName: "istio-data") pod "7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" (UID: "7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:31:41.616689 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.616664 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-token" (OuterVolumeSpecName: "istio-token") pod "7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" (UID: "7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:31:41.616795 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.616685 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" (UID: "7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:31:41.616795 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.616748 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-kube-api-access-xcxvf" (OuterVolumeSpecName: "kube-api-access-xcxvf") pod "7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" (UID: "7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2"). InnerVolumeSpecName "kube-api-access-xcxvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:31:41.616795 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.616769 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" (UID: "7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:31:41.715787 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.715720 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-data\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:41.715787 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.715745 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-podinfo\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:41.715787 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.715755 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-workload-certs\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:41.715787 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.715764 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istiod-ca-cert\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:41.715787 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.715772 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-credential-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:41.715787 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.715781 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-token\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:41.715787 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.715789 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcxvf\" (UniqueName: \"kubernetes.io/projected/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-kube-api-access-xcxvf\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:41.716092 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:41.715799 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2-istio-envoy\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:31:42.298484 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:42.298443 2569 generic.go:356] "Generic (PLEG): container finished" podID="7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" containerID="54a4a34539acb37f5656a67009e8f53bb6f01a4b81bee56478642d44ab44123d" exitCode=0 Jul 13 22:31:42.298681 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:42.298512 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" Jul 13 22:31:42.298681 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:42.298518 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" event={"ID":"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2","Type":"ContainerDied","Data":"54a4a34539acb37f5656a67009e8f53bb6f01a4b81bee56478642d44ab44123d"} Jul 13 22:31:42.298681 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:42.298593 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w" event={"ID":"7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2","Type":"ContainerDied","Data":"e0facbfe60b74a11492f0c6bfce402b4db6ac9f772203d327e6ec691b8d42046"} Jul 13 22:31:42.298681 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:42.298618 2569 scope.go:117] "RemoveContainer" containerID="54a4a34539acb37f5656a67009e8f53bb6f01a4b81bee56478642d44ab44123d" Jul 13 22:31:42.300484 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:42.300455 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" event={"ID":"d6a4dacf-39a7-4386-9778-c99b08deb5c3","Type":"ContainerStarted","Data":"e1e899acada9cc9e40d731781bac2f3c9738d7a4a235b534bf2d037a430a9144"} Jul 13 22:31:42.300642 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:42.300491 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" event={"ID":"d6a4dacf-39a7-4386-9778-c99b08deb5c3","Type":"ContainerStarted","Data":"1c69a61b0f353adae0caebadc5926e32de547151eff46d73fc0960705f7dae88"} Jul 13 22:31:42.300642 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:42.300598 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" Jul 13 22:31:42.308167 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:42.308138 2569 scope.go:117] "RemoveContainer" containerID="1ec93cd55f546e0ae4f00189c2219741f2a6d53b882aa4eab22142f5831588ff" Jul 13 22:31:42.317022 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:42.317005 2569 scope.go:117] "RemoveContainer" containerID="54a4a34539acb37f5656a67009e8f53bb6f01a4b81bee56478642d44ab44123d" Jul 13 22:31:42.317308 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:31:42.317285 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54a4a34539acb37f5656a67009e8f53bb6f01a4b81bee56478642d44ab44123d\": container with ID starting with 54a4a34539acb37f5656a67009e8f53bb6f01a4b81bee56478642d44ab44123d not found: ID does not exist" containerID="54a4a34539acb37f5656a67009e8f53bb6f01a4b81bee56478642d44ab44123d" Jul 13 22:31:42.317429 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:42.317313 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a4a34539acb37f5656a67009e8f53bb6f01a4b81bee56478642d44ab44123d"} err="failed to get container status \"54a4a34539acb37f5656a67009e8f53bb6f01a4b81bee56478642d44ab44123d\": rpc error: code = NotFound desc = could not find container \"54a4a34539acb37f5656a67009e8f53bb6f01a4b81bee56478642d44ab44123d\": container with ID starting with 54a4a34539acb37f5656a67009e8f53bb6f01a4b81bee56478642d44ab44123d not found: ID does not exist" Jul 13 22:31:42.317429 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:42.317330 2569 scope.go:117] "RemoveContainer" containerID="1ec93cd55f546e0ae4f00189c2219741f2a6d53b882aa4eab22142f5831588ff" Jul 13 22:31:42.317616 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:31:42.317583 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ec93cd55f546e0ae4f00189c2219741f2a6d53b882aa4eab22142f5831588ff\": container with ID starting with 1ec93cd55f546e0ae4f00189c2219741f2a6d53b882aa4eab22142f5831588ff not found: ID does not exist" containerID="1ec93cd55f546e0ae4f00189c2219741f2a6d53b882aa4eab22142f5831588ff" Jul 13 22:31:42.317679 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:42.317613 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec93cd55f546e0ae4f00189c2219741f2a6d53b882aa4eab22142f5831588ff"} err="failed to get container status \"1ec93cd55f546e0ae4f00189c2219741f2a6d53b882aa4eab22142f5831588ff\": rpc error: code = NotFound desc = could not find container \"1ec93cd55f546e0ae4f00189c2219741f2a6d53b882aa4eab22142f5831588ff\": container with ID starting with 1ec93cd55f546e0ae4f00189c2219741f2a6d53b882aa4eab22142f5831588ff not found: ID does not exist" Jul 13 22:31:42.345858 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:42.345808 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" podStartSLOduration=1.9044574619999999 podStartE2EDuration="2.345796199s" totalImagesPullingTime="441.338737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:31:42.343288983 +0000 UTC m=+2015.000493752" watchObservedRunningTime="2026-07-13 22:31:42.345796199 +0000 UTC m=+2015.003000966" Jul 13 22:31:42.358627 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:42.358592 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w"] Jul 13 22:31:42.362899 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:42.362876 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d07445-openshift-default-6874b7d469-tp79w"] Jul 13 22:31:43.931255 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:43.931222 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" path="/var/lib/kubelet/pods/7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2/volumes" Jul 13 22:31:45.273476 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.273441 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn"] Jul 13 22:31:45.274109 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.274086 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" containerName="istio-proxy" Jul 13 22:31:45.274182 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.274113 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" containerName="istio-proxy" Jul 13 22:31:45.274285 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.274256 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" containerName="istio-proxy" Jul 13 22:31:45.281593 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.278010 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.295559 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.291902 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-4b5279-openshift-default-dockercfg-mlrz9\"" Jul 13 22:31:45.307562 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.299556 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn"] Jul 13 22:31:45.345285 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.345251 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istiod-ca-cert\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.345450 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.345289 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-podinfo\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.345450 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.345376 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj546\" (UniqueName: \"kubernetes.io/projected/7a98a983-b6c0-4189-994e-9cc1ba7e5383-kube-api-access-mj546\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.345585 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.345445 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-envoy\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.345585 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.345480 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-data\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.345585 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.345500 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-workload-certs\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.345585 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.345570 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-credential-socket\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.345792 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.345589 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-token\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.345792 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.345613 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-workload-socket\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.446219 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.446175 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-workload-socket\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.446219 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.446223 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istiod-ca-cert\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.446428 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.446343 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-podinfo\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.446472 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.446429 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj546\" (UniqueName: \"kubernetes.io/projected/7a98a983-b6c0-4189-994e-9cc1ba7e5383-kube-api-access-mj546\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.446529 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.446513 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-envoy\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.446637 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.446619 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-data\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.446703 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.446642 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-workload-socket\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.446703 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.446652 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-workload-certs\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.446816 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.446714 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-credential-socket\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.446816 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.446743 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-token\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.446928 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.446903 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-workload-certs\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.447045 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.447018 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-credential-socket\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.447045 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.447033 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-data\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.447199 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.447073 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istiod-ca-cert\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.448763 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.448742 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-podinfo\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.448846 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.448787 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-envoy\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.456626 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.456603 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj546\" (UniqueName: \"kubernetes.io/projected/7a98a983-b6c0-4189-994e-9cc1ba7e5383-kube-api-access-mj546\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.456733 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.456693 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-token\") pod \"e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.624675 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.624585 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:45.996518 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:45.996471 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn"] Jul 13 22:31:46.004357 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:31:46.003054 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a98a983_b6c0_4189_994e_9cc1ba7e5383.slice/crio-303b6dcbe9aa5d8b340a0b736b2826536020268eb2ec62a2fb8ef8927a779949 WatchSource:0}: Error finding container 303b6dcbe9aa5d8b340a0b736b2826536020268eb2ec62a2fb8ef8927a779949: Status 404 returned error can't find the container with id 303b6dcbe9aa5d8b340a0b736b2826536020268eb2ec62a2fb8ef8927a779949 Jul 13 22:31:46.006277 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:46.006231 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:31:46.006376 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:46.006331 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:31:46.006436 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:46.006377 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:31:46.330654 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:46.330562 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" event={"ID":"7a98a983-b6c0-4189-994e-9cc1ba7e5383","Type":"ContainerStarted","Data":"475acfec95326fcd3c66687437af4e6334cea21ff22d1ead44ebf952ccd3bacd"} Jul 13 22:31:46.330654 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:46.330634 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" event={"ID":"7a98a983-b6c0-4189-994e-9cc1ba7e5383","Type":"ContainerStarted","Data":"303b6dcbe9aa5d8b340a0b736b2826536020268eb2ec62a2fb8ef8927a779949"} Jul 13 22:31:46.376248 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:46.376201 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" podStartSLOduration=1.3761855330000001 podStartE2EDuration="1.376185533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:31:46.37258067 +0000 UTC m=+2019.029785432" watchObservedRunningTime="2026-07-13 22:31:46.376185533 +0000 UTC m=+2019.033390301" Jul 13 22:31:46.625233 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:46.625136 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:47.630861 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:47.630828 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:48.313754 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.313725 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" Jul 13 22:31:48.343248 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.343213 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:48.344182 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.344157 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:31:48.430196 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.428929 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp"] Jul 13 22:31:48.430196 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.429630 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" containerName="istio-proxy" Jul 13 22:31:48.430196 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.429652 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" containerName="istio-proxy" Jul 13 22:31:48.430196 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.429780 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ac6b5da-7d4b-4617-a8ed-7f818b8aa4b2" containerName="istio-proxy" Jul 13 22:31:48.433714 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.433693 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp"] Jul 13 22:31:48.433868 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.433774 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" Jul 13 22:31:48.436527 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.436425 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-4b5279-serving-cert\"" Jul 13 22:31:48.476160 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.476135 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z8jg\" (UniqueName: \"kubernetes.io/projected/5ce39b53-d250-4e27-99c4-4cbac1e96112-kube-api-access-6z8jg\") pod \"maas-api-e2e-shared-b-4b5279-774f44ff74-864vp\" (UID: \"5ce39b53-d250-4e27-99c4-4cbac1e96112\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" Jul 13 22:31:48.476273 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.476170 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce39b53-d250-4e27-99c4-4cbac1e96112-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4b5279-774f44ff74-864vp\" (UID: \"5ce39b53-d250-4e27-99c4-4cbac1e96112\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" Jul 13 22:31:48.476273 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.476251 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce39b53-d250-4e27-99c4-4cbac1e96112-maas-api-tls\") pod \"maas-api-e2e-shared-b-4b5279-774f44ff74-864vp\" (UID: \"5ce39b53-d250-4e27-99c4-4cbac1e96112\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" Jul 13 22:31:48.577153 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.577072 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce39b53-d250-4e27-99c4-4cbac1e96112-maas-api-tls\") pod \"maas-api-e2e-shared-b-4b5279-774f44ff74-864vp\" (UID: \"5ce39b53-d250-4e27-99c4-4cbac1e96112\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" Jul 13 22:31:48.577300 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.577198 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z8jg\" (UniqueName: \"kubernetes.io/projected/5ce39b53-d250-4e27-99c4-4cbac1e96112-kube-api-access-6z8jg\") pod \"maas-api-e2e-shared-b-4b5279-774f44ff74-864vp\" (UID: \"5ce39b53-d250-4e27-99c4-4cbac1e96112\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" Jul 13 22:31:48.577300 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.577229 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce39b53-d250-4e27-99c4-4cbac1e96112-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4b5279-774f44ff74-864vp\" (UID: \"5ce39b53-d250-4e27-99c4-4cbac1e96112\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" Jul 13 22:31:48.578035 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.578003 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce39b53-d250-4e27-99c4-4cbac1e96112-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4b5279-774f44ff74-864vp\" (UID: \"5ce39b53-d250-4e27-99c4-4cbac1e96112\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" Jul 13 22:31:48.589877 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.589831 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z8jg\" (UniqueName: \"kubernetes.io/projected/5ce39b53-d250-4e27-99c4-4cbac1e96112-kube-api-access-6z8jg\") pod \"maas-api-e2e-shared-b-4b5279-774f44ff74-864vp\" (UID: \"5ce39b53-d250-4e27-99c4-4cbac1e96112\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" Jul 13 22:31:48.591063 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.591020 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce39b53-d250-4e27-99c4-4cbac1e96112-maas-api-tls\") pod \"maas-api-e2e-shared-b-4b5279-774f44ff74-864vp\" (UID: \"5ce39b53-d250-4e27-99c4-4cbac1e96112\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" Jul 13 22:31:48.748832 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.748797 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" Jul 13 22:31:48.937545 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:48.937503 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp"] Jul 13 22:31:48.938669 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:31:48.938646 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce39b53_d250_4e27_99c4_4cbac1e96112.slice/crio-82ced1b2dcdb0f6e37ecd884d7be81834194b7581f6415267a8a0c0f32669463 WatchSource:0}: Error finding container 82ced1b2dcdb0f6e37ecd884d7be81834194b7581f6415267a8a0c0f32669463: Status 404 returned error can't find the container with id 82ced1b2dcdb0f6e37ecd884d7be81834194b7581f6415267a8a0c0f32669463 Jul 13 22:31:49.351376 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:49.351341 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" event={"ID":"5ce39b53-d250-4e27-99c4-4cbac1e96112","Type":"ContainerStarted","Data":"82ced1b2dcdb0f6e37ecd884d7be81834194b7581f6415267a8a0c0f32669463"} Jul 13 22:31:50.358072 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:50.358025 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" event={"ID":"5ce39b53-d250-4e27-99c4-4cbac1e96112","Type":"ContainerStarted","Data":"e1dd93a68a54886554bc8c62188c456db303ec29f6fd181b4854c3e160822fdc"} Jul 13 22:31:50.358616 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:50.358232 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" Jul 13 22:31:50.378371 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:50.378311 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" podStartSLOduration=1.902818616 podStartE2EDuration="2.378288429s" totalImagesPullingTime="475.469813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:31:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:31:50.37406875 +0000 UTC m=+2023.031273521" watchObservedRunningTime="2026-07-13 22:31:50.378288429 +0000 UTC m=+2023.035493197" Jul 13 22:31:52.370729 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:52.370640 2569 generic.go:356] "Generic (PLEG): container finished" podID="d5181cf8-453e-4dd6-a378-1a33a198bdce" containerID="7ea6329adad096bdd129752f75e1472685071c6b73eccbcad8ee8e63605c51d3" exitCode=0 Jul 13 22:31:52.370729 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:52.370715 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" event={"ID":"d5181cf8-453e-4dd6-a378-1a33a198bdce","Type":"ContainerDied","Data":"7ea6329adad096bdd129752f75e1472685071c6b73eccbcad8ee8e63605c51d3"} Jul 13 22:31:52.371159 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:52.371082 2569 scope.go:117] "RemoveContainer" containerID="7ea6329adad096bdd129752f75e1472685071c6b73eccbcad8ee8e63605c51d3" Jul 13 22:31:52.372031 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:52.372006 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:31:52.372093 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:52.372071 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:31:52.372134 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:52.372103 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:31:53.377524 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:53.377491 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" event={"ID":"d5181cf8-453e-4dd6-a378-1a33a198bdce","Type":"ContainerStarted","Data":"1eb0cc5ba6f60483a5d1cd9ee8f3ae2549654ebd2a96c9715015abff919d93df"} Jul 13 22:31:53.494854 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:53.494806 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:53.496310 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:53.496251 2569 patch_prober.go:28] interesting pod/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 13 22:31:53.496471 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:53.496340 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" podUID="d5181cf8-453e-4dd6-a378-1a33a198bdce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 13 22:31:54.280433 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:54.280386 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:54.494956 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:54.494909 2569 patch_prober.go:28] interesting pod/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 13 22:31:54.495467 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:54.494977 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" podUID="d5181cf8-453e-4dd6-a378-1a33a198bdce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 13 22:31:55.494919 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:55.494878 2569 patch_prober.go:28] interesting pod/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 13 22:31:55.495220 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:55.495189 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" podUID="d5181cf8-453e-4dd6-a378-1a33a198bdce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 13 22:31:56.370203 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:56.370172 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" Jul 13 22:31:56.498560 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:56.497998 2569 patch_prober.go:28] interesting pod/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 13 22:31:56.498560 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:56.498083 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" podUID="d5181cf8-453e-4dd6-a378-1a33a198bdce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 13 22:31:57.498676 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:57.498646 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:31:57.499856 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:31:57.499833 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:32:00.427505 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:32:00.427406 2569 generic.go:356] "Generic (PLEG): container finished" podID="7a98a983-b6c0-4189-994e-9cc1ba7e5383" containerID="475acfec95326fcd3c66687437af4e6334cea21ff22d1ead44ebf952ccd3bacd" exitCode=0 Jul 13 22:32:00.427505 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:32:00.427479 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" event={"ID":"7a98a983-b6c0-4189-994e-9cc1ba7e5383","Type":"ContainerDied","Data":"475acfec95326fcd3c66687437af4e6334cea21ff22d1ead44ebf952ccd3bacd"} Jul 13 22:32:00.427951 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:32:00.427902 2569 scope.go:117] "RemoveContainer" containerID="475acfec95326fcd3c66687437af4e6334cea21ff22d1ead44ebf952ccd3bacd" Jul 13 22:32:00.428947 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:32:00.428922 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:32:00.429032 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:32:00.428984 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:32:00.429032 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:32:00.429014 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 22:32:01.435386 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:32:01.435347 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" event={"ID":"7a98a983-b6c0-4189-994e-9cc1ba7e5383","Type":"ContainerStarted","Data":"737e4548645b6644e39476d70ef326dac33ce2faeabe16e492b1fbe613a3f1db"} Jul 13 22:32:01.624960 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:32:01.624922 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:32:01.626361 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:32:01.626330 2569 patch_prober.go:28] interesting pod/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 13 22:32:01.626496 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:32:01.626392 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" podUID="7a98a983-b6c0-4189-994e-9cc1ba7e5383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 13 22:32:02.625865 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:32:02.625829 2569 patch_prober.go:28] interesting pod/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 13 22:32:02.626266 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:32:02.625884 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" podUID="7a98a983-b6c0-4189-994e-9cc1ba7e5383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 13 22:32:03.343776 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:32:03.343742 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:32:03.625214 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:32:03.625123 2569 patch_prober.go:28] interesting pod/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 13 22:32:03.625214 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:32:03.625192 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" podUID="7a98a983-b6c0-4189-994e-9cc1ba7e5383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 13 22:32:04.625202 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:32:04.625142 2569 patch_prober.go:28] interesting pod/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 13 22:32:04.625636 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:32:04.625242 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" podUID="7a98a983-b6c0-4189-994e-9cc1ba7e5383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 13 22:32:05.628481 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:32:05.628452 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:32:05.629607 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:32:05.629587 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:33:07.986462 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:33:07.986428 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 22:33:07.989970 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:33:07.989947 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 22:33:07.997465 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:33:07.997446 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 22:33:08.000715 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:33:08.000692 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 22:36:08.166980 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:36:08.166946 2569 scope.go:117] "RemoveContainer" containerID="b369007214310794374670e989baa3babf31aa69cf8c270c7f6a9c107f410949" Jul 13 22:37:49.965904 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:49.965871 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7556-140665181713328"] Jul 13 22:37:49.969487 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:49.969467 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665181713328" Jul 13 22:37:49.972224 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:49.972185 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-gr4sb\"" Jul 13 22:37:49.976958 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:49.976935 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7556-140665181713328"] Jul 13 22:37:50.045681 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:50.045650 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97f4z\" (UniqueName: \"kubernetes.io/projected/0450e88b-ae4a-4c10-896b-7d14f3b81c68-kube-api-access-97f4z\") pod \"test-curl-7556-140665181713328\" (UID: \"0450e88b-ae4a-4c10-896b-7d14f3b81c68\") " pod="openshift-ingress/test-curl-7556-140665181713328" Jul 13 22:37:50.146759 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:50.146727 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97f4z\" (UniqueName: \"kubernetes.io/projected/0450e88b-ae4a-4c10-896b-7d14f3b81c68-kube-api-access-97f4z\") pod \"test-curl-7556-140665181713328\" (UID: \"0450e88b-ae4a-4c10-896b-7d14f3b81c68\") " pod="openshift-ingress/test-curl-7556-140665181713328" Jul 13 22:37:50.155362 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:50.155334 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97f4z\" (UniqueName: \"kubernetes.io/projected/0450e88b-ae4a-4c10-896b-7d14f3b81c68-kube-api-access-97f4z\") pod \"test-curl-7556-140665181713328\" (UID: \"0450e88b-ae4a-4c10-896b-7d14f3b81c68\") " pod="openshift-ingress/test-curl-7556-140665181713328" Jul 13 22:37:50.280474 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:50.280397 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665181713328" Jul 13 22:37:50.416923 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:50.416897 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7556-140665181713328"] Jul 13 22:37:50.418483 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:37:50.418441 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0450e88b_ae4a_4c10_896b_7d14f3b81c68.slice/crio-9d96d716d4b1c0621faf8bac8857390b42e60f6ebe01fb05665ad0753cc675cd WatchSource:0}: Error finding container 9d96d716d4b1c0621faf8bac8857390b42e60f6ebe01fb05665ad0753cc675cd: Status 404 returned error can't find the container with id 9d96d716d4b1c0621faf8bac8857390b42e60f6ebe01fb05665ad0753cc675cd Jul 13 22:37:50.419827 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:50.419806 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 22:37:51.261773 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:51.261744 2569 generic.go:356] "Generic (PLEG): container finished" podID="0450e88b-ae4a-4c10-896b-7d14f3b81c68" containerID="73763f569b13ed3d6f15a38071a1efe4084ec886aa9edc6094bda225d46c72fa" exitCode=0 Jul 13 22:37:51.262070 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:51.261825 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7556-140665181713328" event={"ID":"0450e88b-ae4a-4c10-896b-7d14f3b81c68","Type":"ContainerDied","Data":"73763f569b13ed3d6f15a38071a1efe4084ec886aa9edc6094bda225d46c72fa"} Jul 13 22:37:51.262070 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:51.261864 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7556-140665181713328" event={"ID":"0450e88b-ae4a-4c10-896b-7d14f3b81c68","Type":"ContainerStarted","Data":"9d96d716d4b1c0621faf8bac8857390b42e60f6ebe01fb05665ad0753cc675cd"} Jul 13 22:37:52.412469 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:52.412443 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665181713328" Jul 13 22:37:52.448576 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:52.448551 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7556-140665181713328_0450e88b-ae4a-4c10-896b-7d14f3b81c68/test-curl-7556-140665181713328/0.log" Jul 13 22:37:52.471633 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:52.471600 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97f4z\" (UniqueName: \"kubernetes.io/projected/0450e88b-ae4a-4c10-896b-7d14f3b81c68-kube-api-access-97f4z\") pod \"0450e88b-ae4a-4c10-896b-7d14f3b81c68\" (UID: \"0450e88b-ae4a-4c10-896b-7d14f3b81c68\") " Jul 13 22:37:52.473923 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:52.473895 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0450e88b-ae4a-4c10-896b-7d14f3b81c68-kube-api-access-97f4z" (OuterVolumeSpecName: "kube-api-access-97f4z") pod "0450e88b-ae4a-4c10-896b-7d14f3b81c68" (UID: "0450e88b-ae4a-4c10-896b-7d14f3b81c68"). InnerVolumeSpecName "kube-api-access-97f4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:37:52.474405 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:52.474384 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7556-140665181713328"] Jul 13 22:37:52.478771 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:52.478747 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7556-140665181713328"] Jul 13 22:37:52.573252 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:52.573179 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97f4z\" (UniqueName: \"kubernetes.io/projected/0450e88b-ae4a-4c10-896b-7d14f3b81c68-kube-api-access-97f4z\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:37:52.595466 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:52.595434 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7556-140665183752752"] Jul 13 22:37:52.595876 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:52.595863 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0450e88b-ae4a-4c10-896b-7d14f3b81c68" containerName="test-curl-7556-140665181713328" Jul 13 22:37:52.595925 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:52.595877 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0450e88b-ae4a-4c10-896b-7d14f3b81c68" containerName="test-curl-7556-140665181713328" Jul 13 22:37:52.595961 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:52.595939 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0450e88b-ae4a-4c10-896b-7d14f3b81c68" containerName="test-curl-7556-140665181713328" Jul 13 22:37:52.599008 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:52.598992 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665183752752" Jul 13 22:37:52.604388 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:52.604366 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7556-140665183752752"] Jul 13 22:37:52.675078 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:52.675044 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g4f9\" (UniqueName: \"kubernetes.io/projected/212c6316-7a4c-4012-b43a-561f55fb515d-kube-api-access-2g4f9\") pod \"test-curl-7556-140665183752752\" (UID: \"212c6316-7a4c-4012-b43a-561f55fb515d\") " pod="openshift-ingress/test-curl-7556-140665183752752" Jul 13 22:37:52.776387 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:52.776358 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g4f9\" (UniqueName: \"kubernetes.io/projected/212c6316-7a4c-4012-b43a-561f55fb515d-kube-api-access-2g4f9\") pod \"test-curl-7556-140665183752752\" (UID: \"212c6316-7a4c-4012-b43a-561f55fb515d\") " pod="openshift-ingress/test-curl-7556-140665183752752" Jul 13 22:37:52.785733 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:52.785703 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g4f9\" (UniqueName: \"kubernetes.io/projected/212c6316-7a4c-4012-b43a-561f55fb515d-kube-api-access-2g4f9\") pod \"test-curl-7556-140665183752752\" (UID: \"212c6316-7a4c-4012-b43a-561f55fb515d\") " pod="openshift-ingress/test-curl-7556-140665183752752" Jul 13 22:37:52.909802 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:52.909708 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665183752752" Jul 13 22:37:53.043110 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:53.043071 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7556-140665183752752"] Jul 13 22:37:53.045181 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:37:53.045150 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod212c6316_7a4c_4012_b43a_561f55fb515d.slice/crio-bdfba64ff0115102890eaa3e302b778465e0501e99d6c4b22eda36f57ad94901 WatchSource:0}: Error finding container bdfba64ff0115102890eaa3e302b778465e0501e99d6c4b22eda36f57ad94901: Status 404 returned error can't find the container with id bdfba64ff0115102890eaa3e302b778465e0501e99d6c4b22eda36f57ad94901 Jul 13 22:37:53.272873 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:53.272845 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665181713328" Jul 13 22:37:53.272873 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:53.272872 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d96d716d4b1c0621faf8bac8857390b42e60f6ebe01fb05665ad0753cc675cd" Jul 13 22:37:53.274086 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:53.274060 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7556-140665183752752" event={"ID":"212c6316-7a4c-4012-b43a-561f55fb515d","Type":"ContainerStarted","Data":"bdfba64ff0115102890eaa3e302b778465e0501e99d6c4b22eda36f57ad94901"} Jul 13 22:37:53.285596 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:37:53.285568 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7556-140665181713328\" is forbidden: User \"system:node:ip-10-0-137-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-137-183.ec2.internal' and this object" podUID="0450e88b-ae4a-4c10-896b-7d14f3b81c68" pod="openshift-ingress/test-curl-7556-140665181713328" Jul 13 22:37:53.931506 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:53.931470 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0450e88b-ae4a-4c10-896b-7d14f3b81c68" path="/var/lib/kubelet/pods/0450e88b-ae4a-4c10-896b-7d14f3b81c68/volumes" Jul 13 22:37:54.280687 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:54.280652 2569 generic.go:356] "Generic (PLEG): container finished" podID="212c6316-7a4c-4012-b43a-561f55fb515d" containerID="6b97df691e78f94cb4d8651a613088c89be52cdb49b121659ca4382568a87920" exitCode=0 Jul 13 22:37:54.280878 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:54.280737 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7556-140665183752752" event={"ID":"212c6316-7a4c-4012-b43a-561f55fb515d","Type":"ContainerDied","Data":"6b97df691e78f94cb4d8651a613088c89be52cdb49b121659ca4382568a87920"} Jul 13 22:37:55.434510 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:55.434484 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665183752752" Jul 13 22:37:55.458638 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:55.458613 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7556-140665183752752_212c6316-7a4c-4012-b43a-561f55fb515d/test-curl-7556-140665183752752/0.log" Jul 13 22:37:55.482158 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:55.482132 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7556-140665183752752"] Jul 13 22:37:55.486921 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:55.486899 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7556-140665183752752"] Jul 13 22:37:55.504192 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:55.504165 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2g4f9\" (UniqueName: \"kubernetes.io/projected/212c6316-7a4c-4012-b43a-561f55fb515d-kube-api-access-2g4f9\") pod \"212c6316-7a4c-4012-b43a-561f55fb515d\" (UID: \"212c6316-7a4c-4012-b43a-561f55fb515d\") " Jul 13 22:37:55.506191 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:55.506172 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212c6316-7a4c-4012-b43a-561f55fb515d-kube-api-access-2g4f9" (OuterVolumeSpecName: "kube-api-access-2g4f9") pod "212c6316-7a4c-4012-b43a-561f55fb515d" (UID: "212c6316-7a4c-4012-b43a-561f55fb515d"). InnerVolumeSpecName "kube-api-access-2g4f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:37:55.605375 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:55.605299 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g4f9\" (UniqueName: \"kubernetes.io/projected/212c6316-7a4c-4012-b43a-561f55fb515d-kube-api-access-2g4f9\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:37:55.641913 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:55.641885 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7556-140665183696768"] Jul 13 22:37:55.642307 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:55.642295 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="212c6316-7a4c-4012-b43a-561f55fb515d" containerName="test-curl-7556-140665183752752" Jul 13 22:37:55.642351 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:55.642309 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="212c6316-7a4c-4012-b43a-561f55fb515d" containerName="test-curl-7556-140665183752752" Jul 13 22:37:55.642395 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:55.642386 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="212c6316-7a4c-4012-b43a-561f55fb515d" containerName="test-curl-7556-140665183752752" Jul 13 22:37:55.645425 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:55.645408 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665183696768" Jul 13 22:37:55.653075 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:55.653055 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7556-140665183696768"] Jul 13 22:37:55.706157 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:55.706131 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7zlr\" (UniqueName: \"kubernetes.io/projected/883b80bd-ca06-4213-8224-3e5bb9ca0949-kube-api-access-n7zlr\") pod \"test-curl-7556-140665183696768\" (UID: \"883b80bd-ca06-4213-8224-3e5bb9ca0949\") " pod="openshift-ingress/test-curl-7556-140665183696768" Jul 13 22:37:55.806868 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:55.806844 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7zlr\" (UniqueName: \"kubernetes.io/projected/883b80bd-ca06-4213-8224-3e5bb9ca0949-kube-api-access-n7zlr\") pod \"test-curl-7556-140665183696768\" (UID: \"883b80bd-ca06-4213-8224-3e5bb9ca0949\") " pod="openshift-ingress/test-curl-7556-140665183696768" Jul 13 22:37:55.816025 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:55.816001 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7zlr\" (UniqueName: \"kubernetes.io/projected/883b80bd-ca06-4213-8224-3e5bb9ca0949-kube-api-access-n7zlr\") pod \"test-curl-7556-140665183696768\" (UID: \"883b80bd-ca06-4213-8224-3e5bb9ca0949\") " pod="openshift-ingress/test-curl-7556-140665183696768" Jul 13 22:37:55.931779 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:55.931713 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="212c6316-7a4c-4012-b43a-561f55fb515d" path="/var/lib/kubelet/pods/212c6316-7a4c-4012-b43a-561f55fb515d/volumes" Jul 13 22:37:55.955463 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:55.955446 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665183696768" Jul 13 22:37:56.089001 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:56.088973 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7556-140665183696768"] Jul 13 22:37:56.090759 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:37:56.090729 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod883b80bd_ca06_4213_8224_3e5bb9ca0949.slice/crio-504644a79169241fdf22d3e3811076dd93f88aa0aa65e66aa19363fd18866538 WatchSource:0}: Error finding container 504644a79169241fdf22d3e3811076dd93f88aa0aa65e66aa19363fd18866538: Status 404 returned error can't find the container with id 504644a79169241fdf22d3e3811076dd93f88aa0aa65e66aa19363fd18866538 Jul 13 22:37:56.293432 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:56.293397 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7556-140665183696768" event={"ID":"883b80bd-ca06-4213-8224-3e5bb9ca0949","Type":"ContainerStarted","Data":"504644a79169241fdf22d3e3811076dd93f88aa0aa65e66aa19363fd18866538"} Jul 13 22:37:56.294618 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:56.294597 2569 scope.go:117] "RemoveContainer" containerID="6b97df691e78f94cb4d8651a613088c89be52cdb49b121659ca4382568a87920" Jul 13 22:37:56.294737 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:56.294616 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665183752752" Jul 13 22:37:57.303529 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:57.303483 2569 generic.go:356] "Generic (PLEG): container finished" podID="883b80bd-ca06-4213-8224-3e5bb9ca0949" containerID="7f5ee33960bc4978c675880e1f6aefda9300b91c065bc304db964c6e46534e7a" exitCode=0 Jul 13 22:37:57.303930 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:57.303565 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7556-140665183696768" event={"ID":"883b80bd-ca06-4213-8224-3e5bb9ca0949","Type":"ContainerDied","Data":"7f5ee33960bc4978c675880e1f6aefda9300b91c065bc304db964c6e46534e7a"} Jul 13 22:37:58.454249 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:58.454225 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665183696768" Jul 13 22:37:58.474569 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:58.474521 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7556-140665183696768_883b80bd-ca06-4213-8224-3e5bb9ca0949/test-curl-7556-140665183696768/0.log" Jul 13 22:37:58.498598 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:58.498568 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7556-140665183696768"] Jul 13 22:37:58.503492 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:58.503467 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7556-140665183696768"] Jul 13 22:37:58.531760 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:58.531729 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7zlr\" (UniqueName: \"kubernetes.io/projected/883b80bd-ca06-4213-8224-3e5bb9ca0949-kube-api-access-n7zlr\") pod \"883b80bd-ca06-4213-8224-3e5bb9ca0949\" (UID: \"883b80bd-ca06-4213-8224-3e5bb9ca0949\") " Jul 13 22:37:58.533675 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:58.533645 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883b80bd-ca06-4213-8224-3e5bb9ca0949-kube-api-access-n7zlr" (OuterVolumeSpecName: "kube-api-access-n7zlr") pod "883b80bd-ca06-4213-8224-3e5bb9ca0949" (UID: "883b80bd-ca06-4213-8224-3e5bb9ca0949"). InnerVolumeSpecName "kube-api-access-n7zlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:37:58.618705 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:58.618647 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7556-140665183696768"] Jul 13 22:37:58.619100 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:58.619085 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="883b80bd-ca06-4213-8224-3e5bb9ca0949" containerName="test-curl-7556-140665183696768" Jul 13 22:37:58.619100 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:58.619099 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="883b80bd-ca06-4213-8224-3e5bb9ca0949" containerName="test-curl-7556-140665183696768" Jul 13 22:37:58.619221 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:58.619185 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="883b80bd-ca06-4213-8224-3e5bb9ca0949" containerName="test-curl-7556-140665183696768" Jul 13 22:37:58.622376 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:58.622359 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665183696768" Jul 13 22:37:58.627471 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:58.627449 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7556-140665183696768"] Jul 13 22:37:58.633097 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:58.633073 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7zlr\" (UniqueName: \"kubernetes.io/projected/883b80bd-ca06-4213-8224-3e5bb9ca0949-kube-api-access-n7zlr\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:37:58.733893 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:58.733852 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l94dq\" (UniqueName: \"kubernetes.io/projected/76e78e4b-4b30-46a3-a0d1-13afc2b647eb-kube-api-access-l94dq\") pod \"test-curl-7556-140665183696768\" (UID: \"76e78e4b-4b30-46a3-a0d1-13afc2b647eb\") " pod="openshift-ingress/test-curl-7556-140665183696768" Jul 13 22:37:58.834518 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:58.834493 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l94dq\" (UniqueName: \"kubernetes.io/projected/76e78e4b-4b30-46a3-a0d1-13afc2b647eb-kube-api-access-l94dq\") pod \"test-curl-7556-140665183696768\" (UID: \"76e78e4b-4b30-46a3-a0d1-13afc2b647eb\") " pod="openshift-ingress/test-curl-7556-140665183696768" Jul 13 22:37:58.842941 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:58.842920 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l94dq\" (UniqueName: \"kubernetes.io/projected/76e78e4b-4b30-46a3-a0d1-13afc2b647eb-kube-api-access-l94dq\") pod \"test-curl-7556-140665183696768\" (UID: \"76e78e4b-4b30-46a3-a0d1-13afc2b647eb\") " pod="openshift-ingress/test-curl-7556-140665183696768" Jul 13 22:37:58.932909 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:58.932851 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665183696768" Jul 13 22:37:59.069259 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:59.069233 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7556-140665183696768"] Jul 13 22:37:59.071128 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:37:59.071097 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76e78e4b_4b30_46a3_a0d1_13afc2b647eb.slice/crio-ae1e0f76c21cbdba4b92017aac814808314fe28b5ecd69d2e31767ee3ead90a6 WatchSource:0}: Error finding container ae1e0f76c21cbdba4b92017aac814808314fe28b5ecd69d2e31767ee3ead90a6: Status 404 returned error can't find the container with id ae1e0f76c21cbdba4b92017aac814808314fe28b5ecd69d2e31767ee3ead90a6 Jul 13 22:37:59.314572 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:59.314530 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665183696768" Jul 13 22:37:59.314728 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:59.314584 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="504644a79169241fdf22d3e3811076dd93f88aa0aa65e66aa19363fd18866538" Jul 13 22:37:59.315774 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:59.315750 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7556-140665183696768" event={"ID":"76e78e4b-4b30-46a3-a0d1-13afc2b647eb","Type":"ContainerStarted","Data":"ae1e0f76c21cbdba4b92017aac814808314fe28b5ecd69d2e31767ee3ead90a6"} Jul 13 22:37:59.326961 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:59.326925 2569 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7556-140665183696768" oldPodUID="883b80bd-ca06-4213-8224-3e5bb9ca0949" podUID="76e78e4b-4b30-46a3-a0d1-13afc2b647eb" Jul 13 22:37:59.932214 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:37:59.932171 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="883b80bd-ca06-4213-8224-3e5bb9ca0949" path="/var/lib/kubelet/pods/883b80bd-ca06-4213-8224-3e5bb9ca0949/volumes" Jul 13 22:38:00.322471 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:00.322436 2569 generic.go:356] "Generic (PLEG): container finished" podID="76e78e4b-4b30-46a3-a0d1-13afc2b647eb" containerID="a763c92a76bb5755b6de89e892f0163020b7fddfb00b21bbe9f047de04d6e438" exitCode=0 Jul 13 22:38:00.322684 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:00.322518 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7556-140665183696768" event={"ID":"76e78e4b-4b30-46a3-a0d1-13afc2b647eb","Type":"ContainerDied","Data":"a763c92a76bb5755b6de89e892f0163020b7fddfb00b21bbe9f047de04d6e438"} Jul 13 22:38:01.471099 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:01.471070 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665183696768" Jul 13 22:38:01.492500 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:01.492473 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7556-140665183696768_76e78e4b-4b30-46a3-a0d1-13afc2b647eb/test-curl-7556-140665183696768/0.log" Jul 13 22:38:01.515992 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:01.515964 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7556-140665183696768"] Jul 13 22:38:01.518107 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:01.518086 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7556-140665183696768"] Jul 13 22:38:01.559933 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:01.559908 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l94dq\" (UniqueName: \"kubernetes.io/projected/76e78e4b-4b30-46a3-a0d1-13afc2b647eb-kube-api-access-l94dq\") pod \"76e78e4b-4b30-46a3-a0d1-13afc2b647eb\" (UID: \"76e78e4b-4b30-46a3-a0d1-13afc2b647eb\") " Jul 13 22:38:01.562002 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:01.561983 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e78e4b-4b30-46a3-a0d1-13afc2b647eb-kube-api-access-l94dq" (OuterVolumeSpecName: "kube-api-access-l94dq") pod "76e78e4b-4b30-46a3-a0d1-13afc2b647eb" (UID: "76e78e4b-4b30-46a3-a0d1-13afc2b647eb"). InnerVolumeSpecName "kube-api-access-l94dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:38:01.640483 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:01.640391 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7556-140665183697776"] Jul 13 22:38:01.641424 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:01.641400 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76e78e4b-4b30-46a3-a0d1-13afc2b647eb" containerName="test-curl-7556-140665183696768" Jul 13 22:38:01.641424 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:01.641425 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="76e78e4b-4b30-46a3-a0d1-13afc2b647eb" containerName="test-curl-7556-140665183696768" Jul 13 22:38:01.641659 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:01.641633 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="76e78e4b-4b30-46a3-a0d1-13afc2b647eb" containerName="test-curl-7556-140665183696768" Jul 13 22:38:01.647310 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:01.647285 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665183697776" Jul 13 22:38:01.649585 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:01.649506 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7556-140665183697776"] Jul 13 22:38:01.660804 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:01.660782 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l94dq\" (UniqueName: \"kubernetes.io/projected/76e78e4b-4b30-46a3-a0d1-13afc2b647eb-kube-api-access-l94dq\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:01.762092 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:01.762066 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwjlp\" (UniqueName: \"kubernetes.io/projected/c4eeecee-6e98-4a35-ab2f-847f13e5a876-kube-api-access-hwjlp\") pod \"test-curl-7556-140665183697776\" (UID: \"c4eeecee-6e98-4a35-ab2f-847f13e5a876\") " pod="openshift-ingress/test-curl-7556-140665183697776" Jul 13 22:38:01.862978 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:01.862949 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwjlp\" (UniqueName: \"kubernetes.io/projected/c4eeecee-6e98-4a35-ab2f-847f13e5a876-kube-api-access-hwjlp\") pod \"test-curl-7556-140665183697776\" (UID: \"c4eeecee-6e98-4a35-ab2f-847f13e5a876\") " pod="openshift-ingress/test-curl-7556-140665183697776" Jul 13 22:38:01.870525 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:01.870500 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwjlp\" (UniqueName: \"kubernetes.io/projected/c4eeecee-6e98-4a35-ab2f-847f13e5a876-kube-api-access-hwjlp\") pod \"test-curl-7556-140665183697776\" (UID: \"c4eeecee-6e98-4a35-ab2f-847f13e5a876\") " pod="openshift-ingress/test-curl-7556-140665183697776" Jul 13 22:38:01.931745 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:01.931673 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76e78e4b-4b30-46a3-a0d1-13afc2b647eb" path="/var/lib/kubelet/pods/76e78e4b-4b30-46a3-a0d1-13afc2b647eb/volumes" Jul 13 22:38:01.958298 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:01.958272 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665183697776" Jul 13 22:38:02.095708 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:02.095682 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7556-140665183697776"] Jul 13 22:38:02.097224 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:38:02.097187 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4eeecee_6e98_4a35_ab2f_847f13e5a876.slice/crio-63466c7208a86a17c11a50903472e2f73fd916d705e7f8317298939393227018 WatchSource:0}: Error finding container 63466c7208a86a17c11a50903472e2f73fd916d705e7f8317298939393227018: Status 404 returned error can't find the container with id 63466c7208a86a17c11a50903472e2f73fd916d705e7f8317298939393227018 Jul 13 22:38:02.334271 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:02.334225 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665183696768" Jul 13 22:38:02.334459 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:02.334224 2569 scope.go:117] "RemoveContainer" containerID="a763c92a76bb5755b6de89e892f0163020b7fddfb00b21bbe9f047de04d6e438" Jul 13 22:38:02.335597 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:02.335457 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7556-140665183697776" event={"ID":"c4eeecee-6e98-4a35-ab2f-847f13e5a876","Type":"ContainerStarted","Data":"63466c7208a86a17c11a50903472e2f73fd916d705e7f8317298939393227018"} Jul 13 22:38:03.342102 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:03.342067 2569 generic.go:356] "Generic (PLEG): container finished" podID="c4eeecee-6e98-4a35-ab2f-847f13e5a876" containerID="2c9b641a3f5e8d11394a4072ad8a20541efa163ff7ebd075855fd59562a401c1" exitCode=0 Jul 13 22:38:03.342496 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:03.342158 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7556-140665183697776" event={"ID":"c4eeecee-6e98-4a35-ab2f-847f13e5a876","Type":"ContainerDied","Data":"2c9b641a3f5e8d11394a4072ad8a20541efa163ff7ebd075855fd59562a401c1"} Jul 13 22:38:04.490414 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:04.490385 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665183697776" Jul 13 22:38:04.509877 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:04.509851 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7556-140665183697776_c4eeecee-6e98-4a35-ab2f-847f13e5a876/test-curl-7556-140665183697776/0.log" Jul 13 22:38:04.531994 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:04.531958 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7556-140665183697776"] Jul 13 22:38:04.535208 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:04.535181 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7556-140665183697776"] Jul 13 22:38:04.587951 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:04.587922 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwjlp\" (UniqueName: \"kubernetes.io/projected/c4eeecee-6e98-4a35-ab2f-847f13e5a876-kube-api-access-hwjlp\") pod \"c4eeecee-6e98-4a35-ab2f-847f13e5a876\" (UID: \"c4eeecee-6e98-4a35-ab2f-847f13e5a876\") " Jul 13 22:38:04.589992 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:04.589958 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4eeecee-6e98-4a35-ab2f-847f13e5a876-kube-api-access-hwjlp" (OuterVolumeSpecName: "kube-api-access-hwjlp") pod "c4eeecee-6e98-4a35-ab2f-847f13e5a876" (UID: "c4eeecee-6e98-4a35-ab2f-847f13e5a876"). InnerVolumeSpecName "kube-api-access-hwjlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:38:04.688884 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:04.688851 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwjlp\" (UniqueName: \"kubernetes.io/projected/c4eeecee-6e98-4a35-ab2f-847f13e5a876-kube-api-access-hwjlp\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:05.354112 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:05.354084 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63466c7208a86a17c11a50903472e2f73fd916d705e7f8317298939393227018" Jul 13 22:38:05.354112 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:05.354084 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7556-140665183697776" Jul 13 22:38:05.363973 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:38:05.363944 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7556-140665183697776\" is forbidden: User \"system:node:ip-10-0-137-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-137-183.ec2.internal' and this object" podUID="c4eeecee-6e98-4a35-ab2f-847f13e5a876" pod="openshift-ingress/test-curl-7556-140665183697776" Jul 13 22:38:05.932291 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:05.932260 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4eeecee-6e98-4a35-ab2f-847f13e5a876" path="/var/lib/kubelet/pods/c4eeecee-6e98-4a35-ab2f-847f13e5a876/volumes" Jul 13 22:38:08.022905 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:08.022878 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 22:38:08.026277 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:08.026254 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 22:38:08.031842 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:08.031825 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 22:38:08.034674 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:08.034656 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jggp_d11fbab3-afd4-4eaa-9b10-c0262d17fc6d/ovn-acl-logging/0.log" Jul 13 22:38:08.133341 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:08.133304 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf"] Jul 13 22:38:08.133607 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:08.133572 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" podUID="d6a4dacf-39a7-4386-9778-c99b08deb5c3" containerName="maas-api" containerID="cri-o://e1e899acada9cc9e40d731781bac2f3c9738d7a4a235b534bf2d037a430a9144" gracePeriod=30 Jul 13 22:38:08.307820 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:08.307733 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" podUID="d6a4dacf-39a7-4386-9778-c99b08deb5c3" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.74:8443/health\": dial tcp 10.132.0.74:8443: connect: connection refused" Jul 13 22:38:08.371279 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:08.371248 2569 generic.go:356] "Generic (PLEG): container finished" podID="d6a4dacf-39a7-4386-9778-c99b08deb5c3" containerID="e1e899acada9cc9e40d731781bac2f3c9738d7a4a235b534bf2d037a430a9144" exitCode=0 Jul 13 22:38:08.371428 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:08.371283 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" event={"ID":"d6a4dacf-39a7-4386-9778-c99b08deb5c3","Type":"ContainerDied","Data":"e1e899acada9cc9e40d731781bac2f3c9738d7a4a235b534bf2d037a430a9144"} Jul 13 22:38:08.403903 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:08.403881 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" Jul 13 22:38:08.531688 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:08.531657 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4rs4\" (UniqueName: \"kubernetes.io/projected/d6a4dacf-39a7-4386-9778-c99b08deb5c3-kube-api-access-f4rs4\") pod \"d6a4dacf-39a7-4386-9778-c99b08deb5c3\" (UID: \"d6a4dacf-39a7-4386-9778-c99b08deb5c3\") " Jul 13 22:38:08.531842 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:08.531707 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6a4dacf-39a7-4386-9778-c99b08deb5c3-maas-api-tls\") pod \"d6a4dacf-39a7-4386-9778-c99b08deb5c3\" (UID: \"d6a4dacf-39a7-4386-9778-c99b08deb5c3\") " Jul 13 22:38:08.531842 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:08.531779 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a4dacf-39a7-4386-9778-c99b08deb5c3-openshift-service-ca\") pod \"d6a4dacf-39a7-4386-9778-c99b08deb5c3\" (UID: \"d6a4dacf-39a7-4386-9778-c99b08deb5c3\") " Jul 13 22:38:08.532174 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:08.532149 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a4dacf-39a7-4386-9778-c99b08deb5c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d6a4dacf-39a7-4386-9778-c99b08deb5c3" (UID: "d6a4dacf-39a7-4386-9778-c99b08deb5c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:38:08.533748 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:08.533728 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6a4dacf-39a7-4386-9778-c99b08deb5c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d6a4dacf-39a7-4386-9778-c99b08deb5c3" (UID: "d6a4dacf-39a7-4386-9778-c99b08deb5c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:38:08.533801 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:08.533782 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a4dacf-39a7-4386-9778-c99b08deb5c3-kube-api-access-f4rs4" (OuterVolumeSpecName: "kube-api-access-f4rs4") pod "d6a4dacf-39a7-4386-9778-c99b08deb5c3" (UID: "d6a4dacf-39a7-4386-9778-c99b08deb5c3"). InnerVolumeSpecName "kube-api-access-f4rs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:38:08.633503 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:08.633437 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a4dacf-39a7-4386-9778-c99b08deb5c3-openshift-service-ca\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:08.633503 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:08.633462 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4rs4\" (UniqueName: \"kubernetes.io/projected/d6a4dacf-39a7-4386-9778-c99b08deb5c3-kube-api-access-f4rs4\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:08.633503 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:08.633472 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6a4dacf-39a7-4386-9778-c99b08deb5c3-maas-api-tls\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:09.377115 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:09.377086 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" Jul 13 22:38:09.377505 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:09.377125 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf" event={"ID":"d6a4dacf-39a7-4386-9778-c99b08deb5c3","Type":"ContainerDied","Data":"1c69a61b0f353adae0caebadc5926e32de547151eff46d73fc0960705f7dae88"} Jul 13 22:38:09.377505 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:09.377181 2569 scope.go:117] "RemoveContainer" containerID="e1e899acada9cc9e40d731781bac2f3c9738d7a4a235b534bf2d037a430a9144" Jul 13 22:38:09.399759 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:09.399734 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf"] Jul 13 22:38:09.401774 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:09.401754 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0eaefd-74fb95676f-4wtsf"] Jul 13 22:38:09.937834 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:09.937795 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6a4dacf-39a7-4386-9778-c99b08deb5c3" path="/var/lib/kubelet/pods/d6a4dacf-39a7-4386-9778-c99b08deb5c3/volumes" Jul 13 22:38:14.482372 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:14.482333 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t"] Jul 13 22:38:14.482870 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:14.482587 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" podUID="d5181cf8-453e-4dd6-a378-1a33a198bdce" containerName="istio-proxy" containerID="cri-o://1eb0cc5ba6f60483a5d1cd9ee8f3ae2549654ebd2a96c9715015abff919d93df" gracePeriod=30 Jul 13 22:38:15.597217 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:15.597185 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp"] Jul 13 22:38:15.597622 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:15.597408 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" podUID="5ce39b53-d250-4e27-99c4-4cbac1e96112" containerName="maas-api" containerID="cri-o://e1dd93a68a54886554bc8c62188c456db303ec29f6fd181b4854c3e160822fdc" gracePeriod=30 Jul 13 22:38:15.878732 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:15.878703 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" Jul 13 22:38:15.897661 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:15.897636 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce39b53-d250-4e27-99c4-4cbac1e96112-openshift-service-ca\") pod \"5ce39b53-d250-4e27-99c4-4cbac1e96112\" (UID: \"5ce39b53-d250-4e27-99c4-4cbac1e96112\") " Jul 13 22:38:15.897811 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:15.897736 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z8jg\" (UniqueName: \"kubernetes.io/projected/5ce39b53-d250-4e27-99c4-4cbac1e96112-kube-api-access-6z8jg\") pod \"5ce39b53-d250-4e27-99c4-4cbac1e96112\" (UID: \"5ce39b53-d250-4e27-99c4-4cbac1e96112\") " Jul 13 22:38:15.898263 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:15.897949 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce39b53-d250-4e27-99c4-4cbac1e96112-maas-api-tls\") pod \"5ce39b53-d250-4e27-99c4-4cbac1e96112\" (UID: \"5ce39b53-d250-4e27-99c4-4cbac1e96112\") " Jul 13 22:38:15.898263 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:15.898030 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce39b53-d250-4e27-99c4-4cbac1e96112-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5ce39b53-d250-4e27-99c4-4cbac1e96112" (UID: "5ce39b53-d250-4e27-99c4-4cbac1e96112"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:38:15.898425 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:15.898345 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce39b53-d250-4e27-99c4-4cbac1e96112-openshift-service-ca\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:15.900268 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:15.900244 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce39b53-d250-4e27-99c4-4cbac1e96112-kube-api-access-6z8jg" (OuterVolumeSpecName: "kube-api-access-6z8jg") pod "5ce39b53-d250-4e27-99c4-4cbac1e96112" (UID: "5ce39b53-d250-4e27-99c4-4cbac1e96112"). InnerVolumeSpecName "kube-api-access-6z8jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:38:15.900372 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:15.900271 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ce39b53-d250-4e27-99c4-4cbac1e96112-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5ce39b53-d250-4e27-99c4-4cbac1e96112" (UID: "5ce39b53-d250-4e27-99c4-4cbac1e96112"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:38:15.999412 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:15.999377 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z8jg\" (UniqueName: \"kubernetes.io/projected/5ce39b53-d250-4e27-99c4-4cbac1e96112-kube-api-access-6z8jg\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:15.999412 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:15.999408 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce39b53-d250-4e27-99c4-4cbac1e96112-maas-api-tls\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:16.416998 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:16.416965 2569 generic.go:356] "Generic (PLEG): container finished" podID="5ce39b53-d250-4e27-99c4-4cbac1e96112" containerID="e1dd93a68a54886554bc8c62188c456db303ec29f6fd181b4854c3e160822fdc" exitCode=0 Jul 13 22:38:16.417177 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:16.417006 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" event={"ID":"5ce39b53-d250-4e27-99c4-4cbac1e96112","Type":"ContainerDied","Data":"e1dd93a68a54886554bc8c62188c456db303ec29f6fd181b4854c3e160822fdc"} Jul 13 22:38:16.417177 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:16.417009 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" Jul 13 22:38:16.417177 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:16.417047 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp" event={"ID":"5ce39b53-d250-4e27-99c4-4cbac1e96112","Type":"ContainerDied","Data":"82ced1b2dcdb0f6e37ecd884d7be81834194b7581f6415267a8a0c0f32669463"} Jul 13 22:38:16.417177 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:16.417070 2569 scope.go:117] "RemoveContainer" containerID="e1dd93a68a54886554bc8c62188c456db303ec29f6fd181b4854c3e160822fdc" Jul 13 22:38:16.426724 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:16.426704 2569 scope.go:117] "RemoveContainer" containerID="e1dd93a68a54886554bc8c62188c456db303ec29f6fd181b4854c3e160822fdc" Jul 13 22:38:16.426978 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:38:16.426956 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1dd93a68a54886554bc8c62188c456db303ec29f6fd181b4854c3e160822fdc\": container with ID starting with e1dd93a68a54886554bc8c62188c456db303ec29f6fd181b4854c3e160822fdc not found: ID does not exist" containerID="e1dd93a68a54886554bc8c62188c456db303ec29f6fd181b4854c3e160822fdc" Jul 13 22:38:16.427052 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:16.426986 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1dd93a68a54886554bc8c62188c456db303ec29f6fd181b4854c3e160822fdc"} err="failed to get container status \"e1dd93a68a54886554bc8c62188c456db303ec29f6fd181b4854c3e160822fdc\": rpc error: code = NotFound desc = could not find container \"e1dd93a68a54886554bc8c62188c456db303ec29f6fd181b4854c3e160822fdc\": container with ID starting with e1dd93a68a54886554bc8c62188c456db303ec29f6fd181b4854c3e160822fdc not found: ID does not exist" Jul 13 22:38:16.441863 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:16.441838 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp"] Jul 13 22:38:16.446011 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:16.445990 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4b5279-774f44ff74-864vp"] Jul 13 22:38:17.931283 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:17.931242 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ce39b53-d250-4e27-99c4-4cbac1e96112" path="/var/lib/kubelet/pods/5ce39b53-d250-4e27-99c4-4cbac1e96112/volumes" Jul 13 22:38:19.748737 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.748665 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:38:19.833251 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.833221 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-credential-socket\") pod \"d5181cf8-453e-4dd6-a378-1a33a198bdce\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " Jul 13 22:38:19.833419 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.833259 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-workload-certs\") pod \"d5181cf8-453e-4dd6-a378-1a33a198bdce\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " Jul 13 22:38:19.833419 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.833297 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-envoy\") pod \"d5181cf8-453e-4dd6-a378-1a33a198bdce\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " Jul 13 22:38:19.833419 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.833321 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-data\") pod \"d5181cf8-453e-4dd6-a378-1a33a198bdce\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " Jul 13 22:38:19.833419 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.833354 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-workload-socket\") pod \"d5181cf8-453e-4dd6-a378-1a33a198bdce\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " Jul 13 22:38:19.833419 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.833402 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzcfk\" (UniqueName: \"kubernetes.io/projected/d5181cf8-453e-4dd6-a378-1a33a198bdce-kube-api-access-jzcfk\") pod \"d5181cf8-453e-4dd6-a378-1a33a198bdce\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " Jul 13 22:38:19.833723 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.833448 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-token\") pod \"d5181cf8-453e-4dd6-a378-1a33a198bdce\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " Jul 13 22:38:19.833723 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.833507 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-podinfo\") pod \"d5181cf8-453e-4dd6-a378-1a33a198bdce\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " Jul 13 22:38:19.833723 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.833502 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d5181cf8-453e-4dd6-a378-1a33a198bdce" (UID: "d5181cf8-453e-4dd6-a378-1a33a198bdce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:19.833723 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.833571 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d5181cf8-453e-4dd6-a378-1a33a198bdce" (UID: "d5181cf8-453e-4dd6-a378-1a33a198bdce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:19.833723 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.833579 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5181cf8-453e-4dd6-a378-1a33a198bdce-istiod-ca-cert\") pod \"d5181cf8-453e-4dd6-a378-1a33a198bdce\" (UID: \"d5181cf8-453e-4dd6-a378-1a33a198bdce\") " Jul 13 22:38:19.833970 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.833726 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d5181cf8-453e-4dd6-a378-1a33a198bdce" (UID: "d5181cf8-453e-4dd6-a378-1a33a198bdce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:19.833970 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.833964 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-credential-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:19.834083 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.833984 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-workload-certs\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:19.834083 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.833999 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-workload-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:19.834083 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.834053 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5181cf8-453e-4dd6-a378-1a33a198bdce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d5181cf8-453e-4dd6-a378-1a33a198bdce" (UID: "d5181cf8-453e-4dd6-a378-1a33a198bdce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:38:19.834308 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.834290 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-data" (OuterVolumeSpecName: "istio-data") pod "d5181cf8-453e-4dd6-a378-1a33a198bdce" (UID: "d5181cf8-453e-4dd6-a378-1a33a198bdce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:19.836125 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.836100 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d5181cf8-453e-4dd6-a378-1a33a198bdce" (UID: "d5181cf8-453e-4dd6-a378-1a33a198bdce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:19.836232 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.836122 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5181cf8-453e-4dd6-a378-1a33a198bdce-kube-api-access-jzcfk" (OuterVolumeSpecName: "kube-api-access-jzcfk") pod "d5181cf8-453e-4dd6-a378-1a33a198bdce" (UID: "d5181cf8-453e-4dd6-a378-1a33a198bdce"). InnerVolumeSpecName "kube-api-access-jzcfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:38:19.836290 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.836244 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d5181cf8-453e-4dd6-a378-1a33a198bdce" (UID: "d5181cf8-453e-4dd6-a378-1a33a198bdce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:38:19.836290 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.836252 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-token" (OuterVolumeSpecName: "istio-token") pod "d5181cf8-453e-4dd6-a378-1a33a198bdce" (UID: "d5181cf8-453e-4dd6-a378-1a33a198bdce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:38:19.934790 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.934766 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-envoy\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:19.934790 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.934786 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-data\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:19.934919 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.934795 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzcfk\" (UniqueName: \"kubernetes.io/projected/d5181cf8-453e-4dd6-a378-1a33a198bdce-kube-api-access-jzcfk\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:19.934919 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.934804 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-token\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:19.934919 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.934813 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5181cf8-453e-4dd6-a378-1a33a198bdce-istio-podinfo\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:19.934919 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:19.934820 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5181cf8-453e-4dd6-a378-1a33a198bdce-istiod-ca-cert\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:20.440993 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:20.440960 2569 generic.go:356] "Generic (PLEG): container finished" podID="d5181cf8-453e-4dd6-a378-1a33a198bdce" containerID="1eb0cc5ba6f60483a5d1cd9ee8f3ae2549654ebd2a96c9715015abff919d93df" exitCode=0 Jul 13 22:38:20.441191 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:20.441010 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" Jul 13 22:38:20.441191 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:20.441040 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" event={"ID":"d5181cf8-453e-4dd6-a378-1a33a198bdce","Type":"ContainerDied","Data":"1eb0cc5ba6f60483a5d1cd9ee8f3ae2549654ebd2a96c9715015abff919d93df"} Jul 13 22:38:20.441191 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:20.441100 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t" event={"ID":"d5181cf8-453e-4dd6-a378-1a33a198bdce","Type":"ContainerDied","Data":"49c3d7186c99e7d34978cebfd887ab2c058bd7d7c6b8475bbacaffab56355aad"} Jul 13 22:38:20.441191 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:20.441130 2569 scope.go:117] "RemoveContainer" containerID="1eb0cc5ba6f60483a5d1cd9ee8f3ae2549654ebd2a96c9715015abff919d93df" Jul 13 22:38:20.450643 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:20.450621 2569 scope.go:117] "RemoveContainer" containerID="7ea6329adad096bdd129752f75e1472685071c6b73eccbcad8ee8e63605c51d3" Jul 13 22:38:20.458330 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:20.458308 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t"] Jul 13 22:38:20.459498 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:20.459484 2569 scope.go:117] "RemoveContainer" containerID="1eb0cc5ba6f60483a5d1cd9ee8f3ae2549654ebd2a96c9715015abff919d93df" Jul 13 22:38:20.459793 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:38:20.459762 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb0cc5ba6f60483a5d1cd9ee8f3ae2549654ebd2a96c9715015abff919d93df\": container with ID starting with 1eb0cc5ba6f60483a5d1cd9ee8f3ae2549654ebd2a96c9715015abff919d93df not found: ID does not exist" containerID="1eb0cc5ba6f60483a5d1cd9ee8f3ae2549654ebd2a96c9715015abff919d93df" Jul 13 22:38:20.459911 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:20.459799 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb0cc5ba6f60483a5d1cd9ee8f3ae2549654ebd2a96c9715015abff919d93df"} err="failed to get container status \"1eb0cc5ba6f60483a5d1cd9ee8f3ae2549654ebd2a96c9715015abff919d93df\": rpc error: code = NotFound desc = could not find container \"1eb0cc5ba6f60483a5d1cd9ee8f3ae2549654ebd2a96c9715015abff919d93df\": container with ID starting with 1eb0cc5ba6f60483a5d1cd9ee8f3ae2549654ebd2a96c9715015abff919d93df not found: ID does not exist" Jul 13 22:38:20.459911 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:20.459816 2569 scope.go:117] "RemoveContainer" containerID="7ea6329adad096bdd129752f75e1472685071c6b73eccbcad8ee8e63605c51d3" Jul 13 22:38:20.460092 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:38:20.460072 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea6329adad096bdd129752f75e1472685071c6b73eccbcad8ee8e63605c51d3\": container with ID starting with 7ea6329adad096bdd129752f75e1472685071c6b73eccbcad8ee8e63605c51d3 not found: ID does not exist" containerID="7ea6329adad096bdd129752f75e1472685071c6b73eccbcad8ee8e63605c51d3" Jul 13 22:38:20.460147 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:20.460102 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea6329adad096bdd129752f75e1472685071c6b73eccbcad8ee8e63605c51d3"} err="failed to get container status \"7ea6329adad096bdd129752f75e1472685071c6b73eccbcad8ee8e63605c51d3\": rpc error: code = NotFound desc = could not find container \"7ea6329adad096bdd129752f75e1472685071c6b73eccbcad8ee8e63605c51d3\": container with ID starting with 7ea6329adad096bdd129752f75e1472685071c6b73eccbcad8ee8e63605c51d3 not found: ID does not exist" Jul 13 22:38:20.464192 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:20.464172 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-0eaefd-openshift-default-699589b9f-vvd2t"] Jul 13 22:38:21.434893 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:21.434856 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn"] Jul 13 22:38:21.435484 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:21.435114 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" podUID="7a98a983-b6c0-4189-994e-9cc1ba7e5383" containerName="istio-proxy" containerID="cri-o://737e4548645b6644e39476d70ef326dac33ce2faeabe16e492b1fbe613a3f1db" gracePeriod=30 Jul 13 22:38:21.931833 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:21.931797 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5181cf8-453e-4dd6-a378-1a33a198bdce" path="/var/lib/kubelet/pods/d5181cf8-453e-4dd6-a378-1a33a198bdce/volumes" Jul 13 22:38:26.699207 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.699180 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:38:26.801568 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.801475 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-workload-certs\") pod \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " Jul 13 22:38:26.801568 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.801531 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj546\" (UniqueName: \"kubernetes.io/projected/7a98a983-b6c0-4189-994e-9cc1ba7e5383-kube-api-access-mj546\") pod \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " Jul 13 22:38:26.801789 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.801595 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-podinfo\") pod \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " Jul 13 22:38:26.801789 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.801611 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-envoy\") pod \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " Jul 13 22:38:26.801789 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.801634 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-credential-socket\") pod \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " Jul 13 22:38:26.801789 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.801661 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istiod-ca-cert\") pod \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " Jul 13 22:38:26.801789 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.801688 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-workload-socket\") pod \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " Jul 13 22:38:26.801789 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.801720 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-token\") pod \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " Jul 13 22:38:26.801789 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.801748 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-data\") pod \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\" (UID: \"7a98a983-b6c0-4189-994e-9cc1ba7e5383\") " Jul 13 22:38:26.802147 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.801819 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7a98a983-b6c0-4189-994e-9cc1ba7e5383" (UID: "7a98a983-b6c0-4189-994e-9cc1ba7e5383"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:26.802147 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.802039 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7a98a983-b6c0-4189-994e-9cc1ba7e5383" (UID: "7a98a983-b6c0-4189-994e-9cc1ba7e5383"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:26.802147 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.802056 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-workload-certs\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:26.802315 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.802226 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7a98a983-b6c0-4189-994e-9cc1ba7e5383" (UID: "7a98a983-b6c0-4189-994e-9cc1ba7e5383"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:38:26.802315 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.802268 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7a98a983-b6c0-4189-994e-9cc1ba7e5383" (UID: "7a98a983-b6c0-4189-994e-9cc1ba7e5383"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:26.802755 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.802734 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-data" (OuterVolumeSpecName: "istio-data") pod "7a98a983-b6c0-4189-994e-9cc1ba7e5383" (UID: "7a98a983-b6c0-4189-994e-9cc1ba7e5383"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:26.803944 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.803913 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7a98a983-b6c0-4189-994e-9cc1ba7e5383" (UID: "7a98a983-b6c0-4189-994e-9cc1ba7e5383"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:26.804038 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.803994 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a98a983-b6c0-4189-994e-9cc1ba7e5383-kube-api-access-mj546" (OuterVolumeSpecName: "kube-api-access-mj546") pod "7a98a983-b6c0-4189-994e-9cc1ba7e5383" (UID: "7a98a983-b6c0-4189-994e-9cc1ba7e5383"). InnerVolumeSpecName "kube-api-access-mj546". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:38:26.804167 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.804144 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-token" (OuterVolumeSpecName: "istio-token") pod "7a98a983-b6c0-4189-994e-9cc1ba7e5383" (UID: "7a98a983-b6c0-4189-994e-9cc1ba7e5383"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:38:26.804260 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.804183 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7a98a983-b6c0-4189-994e-9cc1ba7e5383" (UID: "7a98a983-b6c0-4189-994e-9cc1ba7e5383"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:38:26.902967 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.902924 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj546\" (UniqueName: \"kubernetes.io/projected/7a98a983-b6c0-4189-994e-9cc1ba7e5383-kube-api-access-mj546\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:26.902967 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.902957 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-podinfo\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:26.902967 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.902968 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-envoy\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:26.903233 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.902978 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-credential-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:26.903233 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.902986 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istiod-ca-cert\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:26.903233 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.902995 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-workload-socket\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:26.903233 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.903005 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-token\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:26.903233 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.903013 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a98a983-b6c0-4189-994e-9cc1ba7e5383-istio-data\") on node \"ip-10-0-137-183.ec2.internal\" DevicePath \"\"" Jul 13 22:38:26.953365 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:26.953315 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-d9k4k_3955d6ec-f505-4205-beea-a28c95cc5b64/manager/0.log" Jul 13 22:38:27.072185 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:27.072105 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-qdkhz_e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb/manager/0.log" Jul 13 22:38:27.310227 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:27.310199 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-hfxp7_a592e914-9dce-40aa-9af8-db424b124726/server/0.log" Jul 13 22:38:27.503693 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:27.503662 2569 generic.go:356] "Generic (PLEG): container finished" podID="7a98a983-b6c0-4189-994e-9cc1ba7e5383" containerID="737e4548645b6644e39476d70ef326dac33ce2faeabe16e492b1fbe613a3f1db" exitCode=0 Jul 13 22:38:27.503891 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:27.503708 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" Jul 13 22:38:27.503891 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:27.503745 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" event={"ID":"7a98a983-b6c0-4189-994e-9cc1ba7e5383","Type":"ContainerDied","Data":"737e4548645b6644e39476d70ef326dac33ce2faeabe16e492b1fbe613a3f1db"} Jul 13 22:38:27.503891 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:27.503810 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn" event={"ID":"7a98a983-b6c0-4189-994e-9cc1ba7e5383","Type":"ContainerDied","Data":"303b6dcbe9aa5d8b340a0b736b2826536020268eb2ec62a2fb8ef8927a779949"} Jul 13 22:38:27.503891 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:27.503844 2569 scope.go:117] "RemoveContainer" containerID="737e4548645b6644e39476d70ef326dac33ce2faeabe16e492b1fbe613a3f1db" Jul 13 22:38:27.513389 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:27.513363 2569 scope.go:117] "RemoveContainer" containerID="475acfec95326fcd3c66687437af4e6334cea21ff22d1ead44ebf952ccd3bacd" Jul 13 22:38:27.521375 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:27.521354 2569 scope.go:117] "RemoveContainer" containerID="737e4548645b6644e39476d70ef326dac33ce2faeabe16e492b1fbe613a3f1db" Jul 13 22:38:27.521713 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:38:27.521692 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"737e4548645b6644e39476d70ef326dac33ce2faeabe16e492b1fbe613a3f1db\": container with ID starting with 737e4548645b6644e39476d70ef326dac33ce2faeabe16e492b1fbe613a3f1db not found: ID does not exist" containerID="737e4548645b6644e39476d70ef326dac33ce2faeabe16e492b1fbe613a3f1db" Jul 13 22:38:27.521792 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:27.521718 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"737e4548645b6644e39476d70ef326dac33ce2faeabe16e492b1fbe613a3f1db"} err="failed to get container status \"737e4548645b6644e39476d70ef326dac33ce2faeabe16e492b1fbe613a3f1db\": rpc error: code = NotFound desc = could not find container \"737e4548645b6644e39476d70ef326dac33ce2faeabe16e492b1fbe613a3f1db\": container with ID starting with 737e4548645b6644e39476d70ef326dac33ce2faeabe16e492b1fbe613a3f1db not found: ID does not exist" Jul 13 22:38:27.521792 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:27.521734 2569 scope.go:117] "RemoveContainer" containerID="475acfec95326fcd3c66687437af4e6334cea21ff22d1ead44ebf952ccd3bacd" Jul 13 22:38:27.521974 ip-10-0-137-183 kubenswrapper[2569]: E0713 22:38:27.521957 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475acfec95326fcd3c66687437af4e6334cea21ff22d1ead44ebf952ccd3bacd\": container with ID starting with 475acfec95326fcd3c66687437af4e6334cea21ff22d1ead44ebf952ccd3bacd not found: ID does not exist" containerID="475acfec95326fcd3c66687437af4e6334cea21ff22d1ead44ebf952ccd3bacd" Jul 13 22:38:27.522020 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:27.521978 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475acfec95326fcd3c66687437af4e6334cea21ff22d1ead44ebf952ccd3bacd"} err="failed to get container status \"475acfec95326fcd3c66687437af4e6334cea21ff22d1ead44ebf952ccd3bacd\": rpc error: code = NotFound desc = could not find container \"475acfec95326fcd3c66687437af4e6334cea21ff22d1ead44ebf952ccd3bacd\": container with ID starting with 475acfec95326fcd3c66687437af4e6334cea21ff22d1ead44ebf952ccd3bacd not found: ID does not exist" Jul 13 22:38:27.526171 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:27.526148 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn"] Jul 13 22:38:27.533823 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:27.533802 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-4b5279-openshift-default-5d58c66b5f-x2pnn"] Jul 13 22:38:27.549720 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:27.549702 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787f66975d-44mms_d29f9e2e-e085-4e61-a06f-df808c42f9d1/copy-manifests/0.log" Jul 13 22:38:27.556036 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:27.556013 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787f66975d-44mms_d29f9e2e-e085-4e61-a06f-df808c42f9d1/manager/0.log" Jul 13 22:38:27.937149 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:27.937117 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a98a983-b6c0-4189-994e-9cc1ba7e5383" path="/var/lib/kubelet/pods/7a98a983-b6c0-4189-994e-9cc1ba7e5383/volumes" Jul 13 22:38:28.135259 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:28.135199 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-75cffb6f44-qnk2n_f653da18-9dcf-4511-b761-5f6a5b705c2b/maas-api/0.log" Jul 13 22:38:29.654625 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:29.654584 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-l45vc_712d72a6-274b-4f0f-b4a7-6c2c79820cac/manager/0.log" Jul 13 22:38:29.764188 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:29.764154 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-wwg6g_ef6150b4-e3d0-4c32-bc04-6c491dd31172/manager/0.log" Jul 13 22:38:29.989922 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:29.989893 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-dj6np_2ddaa225-99ef-4b36-b829-b8bae71be5b0/registry-server/0.log" Jul 13 22:38:30.111676 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:30.111647 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-9bbrq_cdda6f89-cd2e-488b-8ebf-26974650d601/manager/0.log" Jul 13 22:38:30.716115 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:30.716085 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4nr29h_58111d7c-bdeb-40f7-b602-c23d9d496066/istio-proxy/0.log" Jul 13 22:38:30.935386 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:30.935354 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-766bd547bb-8trqh_0f150fe9-984a-482a-a204-59a4eb074dba/kube-auth-proxy/0.log" Jul 13 22:38:31.154451 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:31.154334 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cqx9z_f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3/istio-proxy/1.log" Jul 13 22:38:31.528488 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:31.528461 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-768d869684-xh5gb_d793005d-2131-4e53-8054-703b36d050b8/router/0.log" Jul 13 22:38:32.071805 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:32.071777 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-gvtqm_f58bd174-0332-49e3-9b10-51085954d4b8/main/0.log" Jul 13 22:38:36.754123 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754091 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5pw76/must-gather-6dls7"] Jul 13 22:38:36.754508 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754468 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ce39b53-d250-4e27-99c4-4cbac1e96112" containerName="maas-api" Jul 13 22:38:36.754508 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754478 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ce39b53-d250-4e27-99c4-4cbac1e96112" containerName="maas-api" Jul 13 22:38:36.754508 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754491 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6a4dacf-39a7-4386-9778-c99b08deb5c3" containerName="maas-api" Jul 13 22:38:36.754508 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754497 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6a4dacf-39a7-4386-9778-c99b08deb5c3" containerName="maas-api" Jul 13 22:38:36.754508 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754508 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a98a983-b6c0-4189-994e-9cc1ba7e5383" containerName="istio-proxy" Jul 13 22:38:36.754698 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754514 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a98a983-b6c0-4189-994e-9cc1ba7e5383" containerName="istio-proxy" Jul 13 22:38:36.754698 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754529 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5181cf8-453e-4dd6-a378-1a33a198bdce" containerName="istio-proxy" Jul 13 22:38:36.754698 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754547 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5181cf8-453e-4dd6-a378-1a33a198bdce" containerName="istio-proxy" Jul 13 22:38:36.754698 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754554 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a98a983-b6c0-4189-994e-9cc1ba7e5383" containerName="istio-proxy" Jul 13 22:38:36.754698 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754560 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a98a983-b6c0-4189-994e-9cc1ba7e5383" containerName="istio-proxy" Jul 13 22:38:36.754698 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754570 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5181cf8-453e-4dd6-a378-1a33a198bdce" containerName="istio-proxy" Jul 13 22:38:36.754698 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754576 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5181cf8-453e-4dd6-a378-1a33a198bdce" containerName="istio-proxy" Jul 13 22:38:36.754698 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754581 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4eeecee-6e98-4a35-ab2f-847f13e5a876" containerName="test-curl-7556-140665183697776" Jul 13 22:38:36.754698 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754587 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4eeecee-6e98-4a35-ab2f-847f13e5a876" containerName="test-curl-7556-140665183697776" Jul 13 22:38:36.754698 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754642 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ce39b53-d250-4e27-99c4-4cbac1e96112" containerName="maas-api" Jul 13 22:38:36.754698 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754652 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a98a983-b6c0-4189-994e-9cc1ba7e5383" containerName="istio-proxy" Jul 13 22:38:36.754698 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754658 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5181cf8-453e-4dd6-a378-1a33a198bdce" containerName="istio-proxy" Jul 13 22:38:36.754698 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754666 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5181cf8-453e-4dd6-a378-1a33a198bdce" containerName="istio-proxy" Jul 13 22:38:36.754698 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754672 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a98a983-b6c0-4189-994e-9cc1ba7e5383" containerName="istio-proxy" Jul 13 22:38:36.754698 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754678 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4eeecee-6e98-4a35-ab2f-847f13e5a876" containerName="test-curl-7556-140665183697776" Jul 13 22:38:36.754698 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.754684 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6a4dacf-39a7-4386-9778-c99b08deb5c3" containerName="maas-api" Jul 13 22:38:36.759332 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.759310 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5pw76/must-gather-6dls7" Jul 13 22:38:36.762021 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.762001 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5pw76\"/\"openshift-service-ca.crt\"" Jul 13 22:38:36.762138 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.762024 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5pw76\"/\"kube-root-ca.crt\"" Jul 13 22:38:36.763019 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.762997 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5pw76\"/\"default-dockercfg-8z5qb\"" Jul 13 22:38:36.765725 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.765705 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5pw76/must-gather-6dls7"] Jul 13 22:38:36.891546 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.891509 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd6a5763-a80b-4a46-8e64-f4e08f410a62-must-gather-output\") pod \"must-gather-6dls7\" (UID: \"cd6a5763-a80b-4a46-8e64-f4e08f410a62\") " pod="openshift-must-gather-5pw76/must-gather-6dls7" Jul 13 22:38:36.891724 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.891581 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj4dv\" (UniqueName: \"kubernetes.io/projected/cd6a5763-a80b-4a46-8e64-f4e08f410a62-kube-api-access-wj4dv\") pod \"must-gather-6dls7\" (UID: \"cd6a5763-a80b-4a46-8e64-f4e08f410a62\") " pod="openshift-must-gather-5pw76/must-gather-6dls7" Jul 13 22:38:36.992146 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.992111 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd6a5763-a80b-4a46-8e64-f4e08f410a62-must-gather-output\") pod \"must-gather-6dls7\" (UID: \"cd6a5763-a80b-4a46-8e64-f4e08f410a62\") " pod="openshift-must-gather-5pw76/must-gather-6dls7" Jul 13 22:38:36.992313 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.992159 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj4dv\" (UniqueName: \"kubernetes.io/projected/cd6a5763-a80b-4a46-8e64-f4e08f410a62-kube-api-access-wj4dv\") pod \"must-gather-6dls7\" (UID: \"cd6a5763-a80b-4a46-8e64-f4e08f410a62\") " pod="openshift-must-gather-5pw76/must-gather-6dls7" Jul 13 22:38:36.992484 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:36.992458 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd6a5763-a80b-4a46-8e64-f4e08f410a62-must-gather-output\") pod \"must-gather-6dls7\" (UID: \"cd6a5763-a80b-4a46-8e64-f4e08f410a62\") " pod="openshift-must-gather-5pw76/must-gather-6dls7" Jul 13 22:38:37.000965 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:37.000941 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj4dv\" (UniqueName: \"kubernetes.io/projected/cd6a5763-a80b-4a46-8e64-f4e08f410a62-kube-api-access-wj4dv\") pod \"must-gather-6dls7\" (UID: \"cd6a5763-a80b-4a46-8e64-f4e08f410a62\") " pod="openshift-must-gather-5pw76/must-gather-6dls7" Jul 13 22:38:37.069672 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:37.069592 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5pw76/must-gather-6dls7" Jul 13 22:38:37.408947 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:37.408858 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5pw76/must-gather-6dls7"] Jul 13 22:38:37.412198 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:38:37.412159 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd6a5763_a80b_4a46_8e64_f4e08f410a62.slice/crio-edf28db2984526c2d6ffcc7393f72214cd6a92b58e524ff9a83809938478325b WatchSource:0}: Error finding container edf28db2984526c2d6ffcc7393f72214cd6a92b58e524ff9a83809938478325b: Status 404 returned error can't find the container with id edf28db2984526c2d6ffcc7393f72214cd6a92b58e524ff9a83809938478325b Jul 13 22:38:37.555361 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:37.555324 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5pw76/must-gather-6dls7" event={"ID":"cd6a5763-a80b-4a46-8e64-f4e08f410a62","Type":"ContainerStarted","Data":"edf28db2984526c2d6ffcc7393f72214cd6a92b58e524ff9a83809938478325b"} Jul 13 22:38:38.572208 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:38.572159 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5pw76/must-gather-6dls7" event={"ID":"cd6a5763-a80b-4a46-8e64-f4e08f410a62","Type":"ContainerStarted","Data":"7a1a8231965a21b4a75fb31e4698f252e1d98b7f8103dbf3304bcc541cc0c0f9"} Jul 13 22:38:38.572791 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:38.572767 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5pw76/must-gather-6dls7" event={"ID":"cd6a5763-a80b-4a46-8e64-f4e08f410a62","Type":"ContainerStarted","Data":"9dabff7f7eef0dd41f0626e8c48d3b6c070a497769a72d2c6ca9c4b77ced5724"} Jul 13 22:38:38.592888 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:38.592830 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5pw76/must-gather-6dls7" podStartSLOduration=1.880617627 podStartE2EDuration="2.592812261s" totalImagesPullingTime="712.194634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:38:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:38:38.592040662 +0000 UTC m=+2431.249245431" watchObservedRunningTime="2026-07-13 22:38:38.592812261 +0000 UTC m=+2431.250017031" Jul 13 22:38:39.849639 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:39.849608 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qv2xs_2b1adc79-d0de-44ab-b9c5-747c579b20cb/global-pull-secret-syncer/0.log" Jul 13 22:38:39.970570 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:39.970522 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vpg2f_3cc21ccb-5c84-4ed1-8c9e-409a9cc36fa0/konnectivity-agent/0.log" Jul 13 22:38:40.022935 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:40.022898 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-183.ec2.internal_645e95dea7e9a87e47a32d256b790eae/haproxy/0.log" Jul 13 22:38:44.878354 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:44.874332 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-l45vc_712d72a6-274b-4f0f-b4a7-6c2c79820cac/manager/0.log" Jul 13 22:38:44.914003 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:44.913969 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-wwg6g_ef6150b4-e3d0-4c32-bc04-6c491dd31172/manager/0.log" Jul 13 22:38:45.002608 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:45.002572 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-dj6np_2ddaa225-99ef-4b36-b829-b8bae71be5b0/registry-server/0.log" Jul 13 22:38:45.133314 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:45.132965 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-9bbrq_cdda6f89-cd2e-488b-8ebf-26974650d601/manager/0.log" Jul 13 22:38:46.935393 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:46.935363 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-stpjr_779fd208-c240-4e2e-8eed-e76cd9a19023/cluster-monitoring-operator/0.log" Jul 13 22:38:47.308915 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:47.308827 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ptsjg_e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa/node-exporter/0.log" Jul 13 22:38:47.331524 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:47.331488 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ptsjg_e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa/kube-rbac-proxy/0.log" Jul 13 22:38:47.353281 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:47.353249 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ptsjg_e595f8f1-bc53-4bf3-9ba8-815c3d53ecfa/init-textfile/0.log" Jul 13 22:38:47.651142 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:47.651060 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-6pqh9_557b4511-3dad-4e13-92cb-0d5c4eef6af5/prometheus-operator-admission-webhook/0.log" Jul 13 22:38:49.097818 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.097788 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-ddfdp_cf89b421-149a-4fa2-9349-b8b7f2493eaa/networking-console-plugin/0.log" Jul 13 22:38:49.586806 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.586761 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn"] Jul 13 22:38:49.593239 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.593207 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" Jul 13 22:38:49.597760 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.597731 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn"] Jul 13 22:38:49.725783 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.725739 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ddcf9ac5-74d2-4f23-b38f-b82cd39e4771-lib-modules\") pod \"perf-node-gather-daemonset-pz2gn\" (UID: \"ddcf9ac5-74d2-4f23-b38f-b82cd39e4771\") " pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" Jul 13 22:38:49.726043 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.725870 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rdbr\" (UniqueName: \"kubernetes.io/projected/ddcf9ac5-74d2-4f23-b38f-b82cd39e4771-kube-api-access-6rdbr\") pod \"perf-node-gather-daemonset-pz2gn\" (UID: \"ddcf9ac5-74d2-4f23-b38f-b82cd39e4771\") " pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" Jul 13 22:38:49.726043 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.725900 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ddcf9ac5-74d2-4f23-b38f-b82cd39e4771-sys\") pod \"perf-node-gather-daemonset-pz2gn\" (UID: \"ddcf9ac5-74d2-4f23-b38f-b82cd39e4771\") " pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" Jul 13 22:38:49.726043 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.725931 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ddcf9ac5-74d2-4f23-b38f-b82cd39e4771-proc\") pod \"perf-node-gather-daemonset-pz2gn\" (UID: \"ddcf9ac5-74d2-4f23-b38f-b82cd39e4771\") " pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" Jul 13 22:38:49.726043 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.725960 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ddcf9ac5-74d2-4f23-b38f-b82cd39e4771-podres\") pod \"perf-node-gather-daemonset-pz2gn\" (UID: \"ddcf9ac5-74d2-4f23-b38f-b82cd39e4771\") " pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" Jul 13 22:38:49.738327 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.738286 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/1.log" Jul 13 22:38:49.745963 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.745939 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nj7h6_a5e8dab5-94da-46d0-8653-bcc21e849fc1/console-operator/2.log" Jul 13 22:38:49.827091 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.827040 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rdbr\" (UniqueName: \"kubernetes.io/projected/ddcf9ac5-74d2-4f23-b38f-b82cd39e4771-kube-api-access-6rdbr\") pod \"perf-node-gather-daemonset-pz2gn\" (UID: \"ddcf9ac5-74d2-4f23-b38f-b82cd39e4771\") " pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" Jul 13 22:38:49.827288 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.827109 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ddcf9ac5-74d2-4f23-b38f-b82cd39e4771-sys\") pod \"perf-node-gather-daemonset-pz2gn\" (UID: \"ddcf9ac5-74d2-4f23-b38f-b82cd39e4771\") " pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" Jul 13 22:38:49.827288 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.827148 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ddcf9ac5-74d2-4f23-b38f-b82cd39e4771-proc\") pod \"perf-node-gather-daemonset-pz2gn\" (UID: \"ddcf9ac5-74d2-4f23-b38f-b82cd39e4771\") " pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" Jul 13 22:38:49.827288 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.827185 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ddcf9ac5-74d2-4f23-b38f-b82cd39e4771-podres\") pod \"perf-node-gather-daemonset-pz2gn\" (UID: \"ddcf9ac5-74d2-4f23-b38f-b82cd39e4771\") " pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" Jul 13 22:38:49.827288 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.827259 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ddcf9ac5-74d2-4f23-b38f-b82cd39e4771-lib-modules\") pod \"perf-node-gather-daemonset-pz2gn\" (UID: \"ddcf9ac5-74d2-4f23-b38f-b82cd39e4771\") " pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" Jul 13 22:38:49.827525 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.827482 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ddcf9ac5-74d2-4f23-b38f-b82cd39e4771-lib-modules\") pod \"perf-node-gather-daemonset-pz2gn\" (UID: \"ddcf9ac5-74d2-4f23-b38f-b82cd39e4771\") " pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" Jul 13 22:38:49.827930 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.827901 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ddcf9ac5-74d2-4f23-b38f-b82cd39e4771-sys\") pod \"perf-node-gather-daemonset-pz2gn\" (UID: \"ddcf9ac5-74d2-4f23-b38f-b82cd39e4771\") " pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" Jul 13 22:38:49.828011 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.827972 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ddcf9ac5-74d2-4f23-b38f-b82cd39e4771-proc\") pod \"perf-node-gather-daemonset-pz2gn\" (UID: \"ddcf9ac5-74d2-4f23-b38f-b82cd39e4771\") " pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" Jul 13 22:38:49.828075 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.828064 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ddcf9ac5-74d2-4f23-b38f-b82cd39e4771-podres\") pod \"perf-node-gather-daemonset-pz2gn\" (UID: \"ddcf9ac5-74d2-4f23-b38f-b82cd39e4771\") " pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" Jul 13 22:38:49.838599 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.837184 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rdbr\" (UniqueName: \"kubernetes.io/projected/ddcf9ac5-74d2-4f23-b38f-b82cd39e4771-kube-api-access-6rdbr\") pod \"perf-node-gather-daemonset-pz2gn\" (UID: \"ddcf9ac5-74d2-4f23-b38f-b82cd39e4771\") " pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" Jul 13 22:38:49.908493 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:49.908457 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" Jul 13 22:38:50.108731 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:50.108677 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn"] Jul 13 22:38:50.111851 ip-10-0-137-183 kubenswrapper[2569]: W0713 22:38:50.111825 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podddcf9ac5_74d2_4f23_b38f_b82cd39e4771.slice/crio-90e38a366c4aba4bd986bfc33e04127bb141d98435389bfdd6b48d0f191bd4fb WatchSource:0}: Error finding container 90e38a366c4aba4bd986bfc33e04127bb141d98435389bfdd6b48d0f191bd4fb: Status 404 returned error can't find the container with id 90e38a366c4aba4bd986bfc33e04127bb141d98435389bfdd6b48d0f191bd4fb Jul 13 22:38:50.290855 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:50.290823 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-596f749758-2kl94_a1f54888-1879-4e06-9465-d24bb404cce6/console/0.log" Jul 13 22:38:50.323462 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:50.323430 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-78th4_49ff524c-8e50-4f1e-adf4-778791a3191e/download-server/0.log" Jul 13 22:38:50.672330 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:50.672293 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" event={"ID":"ddcf9ac5-74d2-4f23-b38f-b82cd39e4771","Type":"ContainerStarted","Data":"328ad578c89cf3f2ed0fd6226035bd3eca8d86a45e128f04c68dc476d744d550"} Jul 13 22:38:50.672598 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:50.672346 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" event={"ID":"ddcf9ac5-74d2-4f23-b38f-b82cd39e4771","Type":"ContainerStarted","Data":"90e38a366c4aba4bd986bfc33e04127bb141d98435389bfdd6b48d0f191bd4fb"} Jul 13 22:38:50.672598 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:50.672410 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" Jul 13 22:38:50.691406 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:50.691341 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" podStartSLOduration=1.691319958 podStartE2EDuration="1.691319958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 22:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:38:50.689824557 +0000 UTC m=+2443.347029325" watchObservedRunningTime="2026-07-13 22:38:50.691319958 +0000 UTC m=+2443.348524728" Jul 13 22:38:50.900585 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:50.900549 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-sh5jg_2b79086d-97c9-4fe0-a960-a44518765dcf/volume-data-source-validator/0.log" Jul 13 22:38:52.011147 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:52.011057 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9dkjb_f8a4145e-0e8f-43f6-83d7-ef573dbc5929/dns/0.log" Jul 13 22:38:52.035011 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:52.034985 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9dkjb_f8a4145e-0e8f-43f6-83d7-ef573dbc5929/kube-rbac-proxy/0.log" Jul 13 22:38:52.101931 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:52.101904 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4m9ps_f33a38a4-ac69-4c8f-915f-2e84fa701683/dns-node-resolver/0.log" Jul 13 22:38:52.712810 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:52.712762 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-782jn_a16c43cb-24ec-420c-953f-204d3584929c/node-ca/0.log" Jul 13 22:38:53.740302 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:53.740272 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4nr29h_58111d7c-bdeb-40f7-b602-c23d9d496066/istio-proxy/0.log" Jul 13 22:38:53.876671 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:53.876640 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-766bd547bb-8trqh_0f150fe9-984a-482a-a204-59a4eb074dba/kube-auth-proxy/0.log" Jul 13 22:38:53.933022 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:53.932982 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cqx9z_f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3/istio-proxy/0.log" Jul 13 22:38:53.971872 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:53.971836 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cqx9z_f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3/istio-proxy/1.log" Jul 13 22:38:54.059558 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:54.059459 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-768d869684-xh5gb_d793005d-2131-4e53-8054-703b36d050b8/router/0.log" Jul 13 22:38:54.775969 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:54.775936 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-thphz_1fee610d-3565-4cfc-8d40-54ab4e09cac8/serve-healthcheck-canary/0.log" Jul 13 22:38:55.235379 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:55.235343 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-85gwn_3a59df9a-555c-4676-9b35-d1f8b2752e82/insights-operator/0.log" Jul 13 22:38:55.235948 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:55.235928 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-85gwn_3a59df9a-555c-4676-9b35-d1f8b2752e82/insights-operator/1.log" Jul 13 22:38:55.324907 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:55.324869 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bfh2f_abaa946e-e127-4ff9-8072-23c540c62eb5/kube-rbac-proxy/0.log" Jul 13 22:38:55.344187 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:55.344151 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bfh2f_abaa946e-e127-4ff9-8072-23c540c62eb5/exporter/0.log" Jul 13 22:38:55.366403 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:55.366351 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bfh2f_abaa946e-e127-4ff9-8072-23c540c62eb5/extractor/0.log" Jul 13 22:38:56.690015 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:56.689987 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5pw76/perf-node-gather-daemonset-pz2gn" Jul 13 22:38:57.531042 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:57.531013 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-d9k4k_3955d6ec-f505-4205-beea-a28c95cc5b64/manager/0.log" Jul 13 22:38:57.709920 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:57.709887 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-qdkhz_e1ad7a64-ad39-40ca-99b2-f6f05c1cfecb/manager/0.log" Jul 13 22:38:57.841517 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:57.841448 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-hfxp7_a592e914-9dce-40aa-9af8-db424b124726/server/0.log" Jul 13 22:38:57.931657 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:57.931618 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787f66975d-44mms_d29f9e2e-e085-4e61-a06f-df808c42f9d1/manager/0.log" Jul 13 22:38:57.950931 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:38:57.950903 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787f66975d-44mms_d29f9e2e-e085-4e61-a06f-df808c42f9d1/copy-manifests/0.log" Jul 13 22:39:00.590779 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:00.590750 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4nr29h_58111d7c-bdeb-40f7-b602-c23d9d496066/istio-proxy/0.log" Jul 13 22:39:00.706298 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:00.706272 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-766bd547bb-8trqh_0f150fe9-984a-482a-a204-59a4eb074dba/kube-auth-proxy/0.log" Jul 13 22:39:00.737446 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:00.737414 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cqx9z_f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3/istio-proxy/0.log" Jul 13 22:39:00.788622 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:00.788456 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cqx9z_f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3/istio-proxy/1.log" Jul 13 22:39:00.851656 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:00.851575 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-768d869684-xh5gb_d793005d-2131-4e53-8054-703b36d050b8/router/0.log" Jul 13 22:39:02.073919 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:02.073779 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4nr29h_58111d7c-bdeb-40f7-b602-c23d9d496066/istio-proxy/0.log" Jul 13 22:39:02.201697 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:02.201665 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-766bd547bb-8trqh_0f150fe9-984a-482a-a204-59a4eb074dba/kube-auth-proxy/0.log" Jul 13 22:39:02.241321 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:02.241288 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cqx9z_f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3/istio-proxy/0.log" Jul 13 22:39:02.282476 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:02.282448 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cqx9z_f91b6b4b-e207-4557-8a9a-1a4ee92ef4d3/istio-proxy/1.log" Jul 13 22:39:02.350740 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:02.350659 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-768d869684-xh5gb_d793005d-2131-4e53-8054-703b36d050b8/router/0.log" Jul 13 22:39:04.676911 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:04.676881 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-rnlsb_de451fb9-a1ae-4559-86b7-d643a29152df/migrator/0.log" Jul 13 22:39:04.695819 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:04.695792 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-rnlsb_de451fb9-a1ae-4559-86b7-d643a29152df/graceful-termination/0.log" Jul 13 22:39:05.049736 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:05.049695 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-zghwn_8d45c661-160c-4a4e-a7b8-c6e055315ae1/kube-storage-version-migrator-operator/1.log" Jul 13 22:39:05.050699 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:05.050677 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-zghwn_8d45c661-160c-4a4e-a7b8-c6e055315ae1/kube-storage-version-migrator-operator/0.log" Jul 13 22:39:06.200903 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:06.200873 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwskc_83271b6e-2653-44d9-8ddb-03bed0f66e39/kube-multus-additional-cni-plugins/0.log" Jul 13 22:39:06.219639 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:06.219613 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwskc_83271b6e-2653-44d9-8ddb-03bed0f66e39/egress-router-binary-copy/0.log" Jul 13 22:39:06.238494 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:06.238472 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwskc_83271b6e-2653-44d9-8ddb-03bed0f66e39/cni-plugins/0.log" Jul 13 22:39:06.257668 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:06.257642 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwskc_83271b6e-2653-44d9-8ddb-03bed0f66e39/bond-cni-plugin/0.log" Jul 13 22:39:06.279854 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:06.279824 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwskc_83271b6e-2653-44d9-8ddb-03bed0f66e39/routeoverride-cni/0.log" Jul 13 22:39:06.301069 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:06.301048 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwskc_83271b6e-2653-44d9-8ddb-03bed0f66e39/whereabouts-cni-bincopy/0.log" Jul 13 22:39:06.321354 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:06.321329 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwskc_83271b6e-2653-44d9-8ddb-03bed0f66e39/whereabouts-cni/0.log" Jul 13 22:39:06.528120 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:06.528079 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rvqz8_eab93415-1e38-4dbe-9c23-dbc2b0354a12/kube-multus/0.log" Jul 13 22:39:06.642288 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:06.642253 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tj476_227fb8d9-c7c7-4722-bba7-f2c2fc9b482f/network-metrics-daemon/0.log" Jul 13 22:39:06.658379 ip-10-0-137-183 kubenswrapper[2569]: I0713 22:39:06.658354 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tj476_227fb8d9-c7c7-4722-bba7-f2c2fc9b482f/kube-rbac-proxy/0.log"