Jul 06 19:22:33.948050 ip-10-0-140-94 systemd[1]: Starting Kubernetes Kubelet... Jul 06 19:22:35.663810 ip-10-0-140-94 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:22:35.663810 ip-10-0-140-94 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 19:22:35.663810 ip-10-0-140-94 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:22:35.663810 ip-10-0-140-94 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 19:22:35.663810 ip-10-0-140-94 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:22:35.665311 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.665218 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 19:22:35.670856 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670833 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:22:35.670856 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670855 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:22:35.670935 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670860 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:22:35.670935 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670864 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:22:35.670935 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670867 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:22:35.670935 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670870 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:22:35.670935 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670874 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:22:35.670935 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670886 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:22:35.670935 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670890 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:22:35.670935 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670894 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:22:35.670935 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670899 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:22:35.670935 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670903 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:22:35.670935 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670906 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:22:35.670935 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670909 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:22:35.670935 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670912 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:22:35.670935 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670914 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:22:35.670935 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670918 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:22:35.670935 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670920 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:22:35.670935 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670923 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:22:35.670935 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670926 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:22:35.670935 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670929 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670932 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670935 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670938 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670941 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670945 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670948 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670951 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670953 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670956 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670959 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670961 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670964 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670967 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670970 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670972 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670975 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670978 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670982 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670984 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:22:35.671424 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670987 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:22:35.671956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670990 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:22:35.671956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670992 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:22:35.671956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670995 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:22:35.671956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.670999 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:22:35.671956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671002 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:22:35.671956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671006 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:22:35.671956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671009 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:22:35.671956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671012 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:22:35.671956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671015 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:22:35.671956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671018 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:22:35.671956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671021 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:22:35.671956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671024 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:22:35.671956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671027 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:22:35.671956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671031 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:22:35.671956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671034 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:22:35.671956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671037 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:22:35.671956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671039 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:22:35.671956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671042 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:22:35.671956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671045 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:22:35.672503 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671048 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:22:35.672503 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671050 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:22:35.672503 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671053 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:22:35.672503 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671055 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:22:35.672503 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671059 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:22:35.672503 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671061 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:22:35.672503 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671064 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:22:35.672503 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671069 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:22:35.672503 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671072 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:22:35.672503 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671076 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:22:35.672503 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671079 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:22:35.672503 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671082 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:22:35.672503 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671085 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:22:35.672503 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671088 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:22:35.672503 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671091 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:22:35.672503 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671094 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:22:35.672503 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671096 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:22:35.672503 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671099 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:22:35.672503 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671102 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671104 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671107 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671109 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671112 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671115 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671118 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671121 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671125 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671128 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671131 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671133 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671136 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671139 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671141 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671144 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671147 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671150 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671152 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671155 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:22:35.672991 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.671158 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672059 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672072 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672077 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672080 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672084 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672087 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672091 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672094 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672097 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672100 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672102 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672105 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672108 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672111 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672113 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672116 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672119 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672122 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672124 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:22:35.673545 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672127 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:22:35.674111 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672137 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:22:35.674111 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672140 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:22:35.674111 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672142 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:22:35.674111 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672145 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:22:35.674111 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672148 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:22:35.674111 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672153 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:22:35.674111 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672157 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:22:35.674111 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672162 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:22:35.674111 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672165 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:22:35.674111 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672168 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:22:35.674111 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672171 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:22:35.674111 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672174 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:22:35.674111 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672176 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:22:35.674111 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672187 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:22:35.674111 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672191 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:22:35.674111 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672194 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:22:35.674111 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672197 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:22:35.674111 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672199 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672202 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672205 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672209 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672212 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672214 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672217 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672219 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672222 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672225 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672227 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672230 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672233 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672236 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672238 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672241 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672244 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672246 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672249 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672252 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:22:35.674583 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672254 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672257 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672260 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672262 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672265 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672267 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672270 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672273 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672277 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672280 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672283 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672286 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672288 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672293 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672299 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672302 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672305 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672308 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672311 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672314 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:22:35.675139 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672317 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672319 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672322 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672325 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672329 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672334 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672337 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672340 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672343 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672345 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672348 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672351 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672354 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672357 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672360 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672363 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672365 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672368 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672371 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672374 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:22:35.675694 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672377 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672380 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672857 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672864 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672867 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672870 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672873 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672876 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672879 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672882 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672884 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672887 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672890 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672893 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672896 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672899 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672902 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672907 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672910 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672913 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:22:35.676235 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672915 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672918 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672921 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672924 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672926 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672929 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672932 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672935 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672937 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672940 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672943 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672946 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672949 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672952 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672955 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672958 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672961 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672964 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672967 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672969 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:22:35.676751 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672972 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:22:35.677286 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672975 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:22:35.677286 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672977 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:22:35.677286 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672980 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:22:35.677286 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672983 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:22:35.677286 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672988 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:22:35.677286 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672992 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:22:35.677286 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672996 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:22:35.677286 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.672999 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:22:35.677286 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673007 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:22:35.677286 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673010 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:22:35.677286 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673013 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:22:35.677286 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673016 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:22:35.677286 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673019 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:22:35.677286 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673022 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:22:35.677286 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673024 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:22:35.677286 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673027 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:22:35.677286 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673030 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:22:35.677286 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673032 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:22:35.677286 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673035 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:22:35.677787 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673038 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:22:35.677787 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673041 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:22:35.677787 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673043 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:22:35.677787 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673046 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:22:35.677787 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673050 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:22:35.677787 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673053 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:22:35.677787 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673055 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:22:35.677787 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673059 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:22:35.677787 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673062 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:22:35.677787 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673065 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:22:35.677787 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673067 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:22:35.677787 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673070 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:22:35.677787 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673075 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:22:35.677787 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673078 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:22:35.677787 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673081 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:22:35.677787 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673085 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:22:35.677787 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673089 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:22:35.677787 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673092 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:22:35.677787 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673095 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673097 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673100 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673104 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673107 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673110 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673113 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673115 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673118 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673121 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673124 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673127 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673129 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673132 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673134 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673137 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673140 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673143 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673146 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673149 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:22:35.678297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673151 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673154 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673157 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673247 2578 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673256 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673265 2578 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673269 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673274 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673278 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673283 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673287 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673290 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673294 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673298 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673301 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673306 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673310 2578 flags.go:64] FLAG: --cgroup-root="" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673313 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673316 2578 flags.go:64] FLAG: --client-ca-file="" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673319 2578 flags.go:64] FLAG: --cloud-provider="external" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673323 2578 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673327 2578 flags.go:64] FLAG: --cluster-domain="" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673330 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673333 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 19:22:35.678834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673337 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673340 2578 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673344 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673348 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673351 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673354 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673358 2578 flags.go:64] FLAG: --contention-profiling="false" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673361 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673365 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673368 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673372 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673377 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673381 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673384 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673387 2578 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673390 2578 flags.go:64] FLAG: --enable-server="true" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673394 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673399 2578 flags.go:64] FLAG: --event-burst="100" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673402 2578 flags.go:64] FLAG: --event-qps="50" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673405 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673409 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673412 2578 flags.go:64] FLAG: --eviction-hard="" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673416 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673421 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673424 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 19:22:35.679441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673427 2578 flags.go:64] FLAG: --eviction-soft="" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673430 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673433 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673437 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673440 2578 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673443 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673447 2578 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673450 2578 flags.go:64] FLAG: --feature-gates="" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673454 2578 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673458 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673461 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673464 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673467 2578 flags.go:64] FLAG: --healthz-port="10248" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673471 2578 flags.go:64] FLAG: --help="false" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673475 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-140-94.ec2.internal" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673478 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673481 2578 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673485 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673488 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673492 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673496 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673499 2578 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673502 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 19:22:35.680105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673504 2578 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673507 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673511 2578 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673514 2578 flags.go:64] FLAG: --kube-reserved="" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673517 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673520 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673524 2578 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673528 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673531 2578 flags.go:64] FLAG: --lock-file="" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673534 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673537 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673540 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673546 2578 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673549 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673553 2578 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673556 2578 flags.go:64] FLAG: --logging-format="text" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673559 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673563 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673566 2578 flags.go:64] FLAG: --manifest-url="" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673570 2578 flags.go:64] FLAG: --manifest-url-header="" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673575 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673578 2578 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673582 2578 flags.go:64] FLAG: --max-pods="110" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673585 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673589 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 19:22:35.680852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673592 2578 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673595 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673599 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673602 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673606 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673614 2578 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673618 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673621 2578 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673624 2578 flags.go:64] FLAG: --pod-cidr="" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673627 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673634 2578 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673637 2578 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673640 2578 flags.go:64] FLAG: --pods-per-core="0" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673643 2578 flags.go:64] FLAG: --port="10250" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673647 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673651 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06d409865a0e96e38" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673655 2578 flags.go:64] FLAG: --qos-reserved="" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673658 2578 flags.go:64] FLAG: --read-only-port="10255" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673661 2578 flags.go:64] FLAG: --register-node="true" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673664 2578 flags.go:64] FLAG: --register-with-taints="" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673668 2578 flags.go:64] FLAG: --registry-burst="10" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673671 2578 flags.go:64] FLAG: --registry-qps="5" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673675 2578 flags.go:64] FLAG: --reserved-cpus="" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673678 2578 flags.go:64] FLAG: --reserved-memory="" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673682 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 19:22:35.681515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673685 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673692 2578 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673695 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673698 2578 flags.go:64] FLAG: --runonce="false" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673701 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673705 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673708 2578 flags.go:64] FLAG: --seccomp-default="false" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673711 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673714 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673718 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673721 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673725 2578 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673729 2578 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673732 2578 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673735 2578 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673738 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673741 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673744 2578 flags.go:64] FLAG: --system-cgroups="" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673747 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673753 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673756 2578 flags.go:64] FLAG: --tls-cert-file="" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673759 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673763 2578 flags.go:64] FLAG: --tls-min-version="" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673782 2578 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673787 2578 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 19:22:35.682211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673790 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673793 2578 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673797 2578 flags.go:64] FLAG: --v="2" Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673807 2578 flags.go:64] FLAG: --version="false" Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673812 2578 flags.go:64] FLAG: --vmodule="" Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673816 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.673820 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673947 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673953 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673958 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673961 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673964 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673968 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673971 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673974 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673977 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673980 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673983 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673986 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673989 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673993 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:22:35.683165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673996 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.673999 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674001 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674004 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674007 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674010 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674013 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674016 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674019 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674021 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674024 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674027 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674030 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674033 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674036 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674040 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674044 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674048 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674050 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674055 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:22:35.684003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674058 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:22:35.684533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674069 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:22:35.684533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674074 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:22:35.684533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674078 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:22:35.684533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674082 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:22:35.684533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674086 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:22:35.684533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674091 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:22:35.684533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674097 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:22:35.684533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674103 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:22:35.684533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674107 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:22:35.684533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674112 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:22:35.684533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674116 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:22:35.684533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674122 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:22:35.684533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674128 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:22:35.684533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674133 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:22:35.684533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674137 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:22:35.684533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674140 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:22:35.684533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674143 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:22:35.684533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674146 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:22:35.684533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674149 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674152 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674155 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674158 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674160 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674164 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674166 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674169 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674172 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674175 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674177 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674182 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674187 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674191 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674194 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674197 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674200 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674203 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674207 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674210 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:22:35.685051 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674213 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674216 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674218 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674221 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674224 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674227 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674231 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674234 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674236 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674239 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674242 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674244 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674248 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674251 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674254 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674256 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674259 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674262 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674264 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674267 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:22:35.685571 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674270 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:22:35.686249 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674273 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:22:35.686249 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674276 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:22:35.686249 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674278 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:22:35.686249 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674282 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:22:35.686249 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.674285 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:22:35.686249 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.674936 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:22:35.686249 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.681915 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 19:22:35.686249 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.681935 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 19:22:35.686249 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.681956 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 19:22:35.686249 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.681967 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 19:22:35.686249 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682029 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:22:35.686249 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682035 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:22:35.686249 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682039 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:22:35.686249 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682042 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:22:35.686249 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682045 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:22:35.686655 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682049 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:22:35.686655 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682052 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:22:35.686655 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682056 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:22:35.686655 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682061 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:22:35.686655 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682065 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:22:35.686655 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682068 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:22:35.686655 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682071 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:22:35.686655 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682075 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:22:35.686655 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682079 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:22:35.686655 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682082 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:22:35.686655 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682085 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:22:35.686655 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682088 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:22:35.686655 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682091 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:22:35.686655 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682094 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:22:35.686655 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682099 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:22:35.686655 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682102 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:22:35.686655 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682104 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:22:35.686655 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682107 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:22:35.686655 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682110 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682113 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682115 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682119 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682122 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682125 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682128 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682131 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682135 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682137 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682140 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682143 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682146 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682149 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682152 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682155 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682157 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682161 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682163 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682166 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:22:35.687179 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682169 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682171 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682174 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682177 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682180 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682183 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682186 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682189 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682192 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682195 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682198 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682201 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682204 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682206 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682209 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682213 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682216 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682219 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682222 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682225 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:22:35.687709 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682228 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682231 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682234 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682237 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682239 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682242 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682245 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682247 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682250 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682253 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682255 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682258 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682261 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682263 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682267 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682269 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682274 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682278 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682281 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682284 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:22:35.688243 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682287 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:22:35.688756 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682289 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:22:35.688756 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682292 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:22:35.688756 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682295 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:22:35.688756 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682297 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:22:35.688756 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682300 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:22:35.688756 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682304 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:22:35.688756 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682306 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:22:35.688756 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682309 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:22:35.688756 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682314 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:22:35.688756 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682317 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:22:35.688756 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682321 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:22:35.688756 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682324 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:22:35.688756 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682327 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:22:35.688756 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682331 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:22:35.688756 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.682333 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:22:35.689165 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.682339 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:22:35.689165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683545 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:22:35.689165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683561 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:22:35.689165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683573 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:22:35.689165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683578 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:22:35.689165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683584 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:22:35.689165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683589 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:22:35.689165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683595 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:22:35.689165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683601 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:22:35.689165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683609 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:22:35.689165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683616 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:22:35.689165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683622 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:22:35.689165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683627 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:22:35.689165 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683633 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683638 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683650 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683656 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683662 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683669 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683674 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683679 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683684 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683689 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683695 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683700 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683705 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683710 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683721 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683727 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683734 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683739 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683744 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683748 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:22:35.689549 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683754 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683758 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683782 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683788 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683793 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683798 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683803 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683815 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683820 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683825 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683830 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683835 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683840 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683845 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683850 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683855 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683860 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683865 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683870 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683881 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:22:35.690222 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683887 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683891 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683897 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683903 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683908 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683913 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683921 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683927 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683932 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683938 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683942 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683945 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683951 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683954 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683958 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.683961 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684029 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684181 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684188 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684192 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:22:35.690733 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684195 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:22:35.691270 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684198 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:22:35.691270 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684201 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:22:35.691270 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684207 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:22:35.691270 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684212 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:22:35.691270 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684215 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:22:35.691270 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684218 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:22:35.691270 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684221 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:22:35.691270 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684224 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:22:35.691270 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684227 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:22:35.691270 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684231 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:22:35.691270 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684234 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:22:35.691270 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684237 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:22:35.691270 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684240 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:22:35.691270 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684243 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:22:35.691270 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684246 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:22:35.691270 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684249 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:22:35.691270 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684251 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:22:35.691270 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684254 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:22:35.691270 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684258 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:22:35.691786 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684261 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:22:35.691786 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684264 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:22:35.691786 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684268 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:22:35.691786 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684271 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:22:35.691786 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684274 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:22:35.691786 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684276 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:22:35.691786 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:35.684279 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:22:35.691786 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.684286 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:22:35.691786 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.684464 2578 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 19:22:35.691786 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.687309 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 19:22:35.691786 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.688828 2578 server.go:1019] "Starting client certificate rotation" Jul 06 19:22:35.691786 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.688937 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:22:35.691786 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.688975 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:22:35.692492 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.692476 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:22:35.696378 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.696363 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:22:35.708475 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.708447 2578 log.go:25] "Validated CRI v1 runtime API" Jul 06 19:22:35.711867 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.711842 2578 log.go:25] "Validated CRI v1 image API" Jul 06 19:22:35.715252 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.715233 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 19:22:35.717694 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.717673 2578 fs.go:135] Filesystem UUIDs: map[0b84fbad-fb03-4f3b-acf6-9bad354f3eef:/dev/nvme0n1p3 6b2b37c8-4534-419d-bd02-f0d0678d2e30:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 06 19:22:35.717796 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.717693 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 19:22:35.723213 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.723088 2578 manager.go:216] Machine: {Timestamp:2026-07-06 19:22:35.72171455 +0000 UTC m=+0.350668513 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104016 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e20e0493ca7d960051a2d631091c8 SystemUUID:ec2e20e0-493c-a7d9-6005-1a2d631091c8 BootID:c9fe37a5-8e57-467d-979a-86064679bd47 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:79:33:76:00:3f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:79:33:76:00:3f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:3a:46:f4:50:9e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 19:22:35.723213 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.723206 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 19:22:35.723358 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.723296 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 19:22:35.726500 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.726470 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 19:22:35.726659 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.726505 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-94.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 19:22:35.727227 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.727215 2578 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 19:22:35.727270 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.727230 2578 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 19:22:35.727270 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.727238 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:22:35.727860 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.727849 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:22:35.729155 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.729143 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 19:22:35.732594 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.732581 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:22:35.732892 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.732876 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 19:22:35.733710 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.733683 2578 kubelet.go:486] "Attempting to sync node with API server" Jul 06 19:22:35.733710 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.733711 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 19:22:35.733892 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.733732 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 19:22:35.733892 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.733752 2578 kubelet.go:398] "Adding apiserver pod source" Jul 06 19:22:35.733892 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.733783 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 19:22:35.734353 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.734329 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:22:35.734905 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.734887 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:22:35.737481 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.737462 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 19:22:35.739447 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.739429 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 19:22:35.739538 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.739524 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 19:22:35.740564 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.740547 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 19:22:35.740656 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.740570 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 19:22:35.740656 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.740580 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 19:22:35.740656 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.740588 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 19:22:35.740656 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.740601 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 19:22:35.740656 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.740610 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 19:22:35.740656 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.740625 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 19:22:35.740656 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.740634 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 19:22:35.740656 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.740646 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 19:22:35.740656 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.740654 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 19:22:35.740953 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.740667 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 19:22:35.740953 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.740677 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 19:22:35.741394 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.741382 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 19:22:35.741449 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.741397 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 19:22:35.743832 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.743811 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:22:35.745301 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.745283 2578 server.go:1268] "Started kubelet" Jul 06 19:22:35.745424 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.745398 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 19:22:35.745507 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.745467 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 19:22:35.745553 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.745529 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 19:22:35.747160 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.747142 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 19:22:35.748618 ip-10-0-140-94 systemd[1]: Started Kubernetes Kubelet. Jul 06 19:22:35.749852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.749837 2578 server.go:310] "Adding debug handlers to kubelet server" Jul 06 19:22:35.751020 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.750973 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 19:22:35.752234 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.752214 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-94.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 19:22:35.753431 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.752515 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c7caaecc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.74525102 +0000 UTC m=+0.374204979,LastTimestamp:2026-07-06 19:22:35.74525102 +0000 UTC m=+0.374204979,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:35.753868 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.753847 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 19:22:35.753868 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.753853 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 19:22:35.754037 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.753967 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found" Jul 06 19:22:35.754475 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.754453 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 19:22:35.754475 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.754455 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 19:22:35.754622 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.754487 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 19:22:35.754622 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.754589 2578 reconstruct.go:97] "Volume reconstruction finished" Jul 06 19:22:35.754622 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.754600 2578 reconciler.go:29] "Reconciler: start to sync state" Jul 06 19:22:35.754976 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.754959 2578 factory.go:153] Registering CRI-O factory Jul 06 19:22:35.754976 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.754978 2578 factory.go:223] Registration of the crio container factory successfully Jul 06 19:22:35.755125 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.755049 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 19:22:35.755125 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.755060 2578 factory.go:55] Registering systemd factory Jul 06 19:22:35.755125 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.755067 2578 factory.go:223] Registration of the systemd container factory successfully Jul 06 19:22:35.755125 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.755083 2578 factory.go:103] Registering Raw factory Jul 06 19:22:35.755125 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.755096 2578 manager.go:1196] Started watching for new ooms in manager Jul 06 19:22:35.755684 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.755667 2578 manager.go:349] Starting recovery of all containers Jul 06 19:22:35.760891 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.760856 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 19:22:35.761002 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.760902 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 19:22:35.761097 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.761066 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 19:22:35.761186 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.761162 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 19:22:35.768320 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.768298 2578 manager.go:354] Recovery completed Jul 06 19:22:35.772931 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.772913 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:22:35.775451 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.775432 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:22:35.775536 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.775467 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:22:35.775536 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.775481 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:22:35.776139 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.776119 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 19:22:35.776139 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.776131 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:22:35.776139 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.776138 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 19:22:35.776320 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.776156 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:22:35.776965 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.776950 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:22:35.777036 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.776978 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:22:35.777036 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.776994 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:22:35.777262 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.777250 2578 policy_none.go:49] "None policy: Start" Jul 06 19:22:35.777307 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.777267 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 19:22:35.777307 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.777279 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 19:22:35.778211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.778200 2578 policy_none.go:47] "Start" Jul 06 19:22:35.778573 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.778502 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c9977e80 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-94.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.775450752 +0000 UTC m=+0.404404712,LastTimestamp:2026-07-06 19:22:35.775450752 +0000 UTC m=+0.404404712,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:35.791264 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.791189 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c997db07 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-94.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.775474439 +0000 UTC m=+0.404428399,LastTimestamp:2026-07-06 19:22:35.775474439 +0000 UTC m=+0.404428399,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:35.799955 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.799874 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c99804fa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-94.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.775485178 +0000 UTC m=+0.404439139,LastTimestamp:2026-07-06 19:22:35.775485178 +0000 UTC m=+0.404439139,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:35.810063 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.809983 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-94.ec2.internal.18bfc9b4c9977e80\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c9977e80 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-94.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.775450752 +0000 UTC m=+0.404404712,LastTimestamp:2026-07-06 19:22:35.776964254 +0000 UTC m=+0.405918214,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:35.814487 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.814470 2578 manager.go:337] "Starting Device Plugin manager" Jul 06 19:22:35.814564 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.814512 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 19:22:35.814564 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.814524 2578 server.go:87] "Starting device plugin registration server" Jul 06 19:22:35.814843 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.814828 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 19:22:35.814921 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.814844 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 19:22:35.814967 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.814931 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 19:22:35.815045 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.815031 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 19:22:35.815093 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.815045 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 19:22:35.815635 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.815615 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 19:22:35.815740 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.815667 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-94.ec2.internal\" not found" Jul 06 19:22:35.821076 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.821006 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-94.ec2.internal.18bfc9b4c997db07\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c997db07 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-94.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.775474439 +0000 UTC m=+0.404428399,LastTimestamp:2026-07-06 19:22:35.776982756 +0000 UTC m=+0.405936716,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:35.827228 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.827148 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-94.ec2.internal.18bfc9b4c99804fa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c99804fa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-94.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.775485178 +0000 UTC m=+0.404439139,LastTimestamp:2026-07-06 19:22:35.777000379 +0000 UTC m=+0.405954338,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:35.843025 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.842928 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4cc0fbed1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.816885969 +0000 UTC m=+0.445839919,LastTimestamp:2026-07-06 19:22:35.816885969 +0000 UTC m=+0.445839919,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:35.867200 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.867165 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 19:22:35.868567 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.868540 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 19:22:35.868634 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.868600 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 19:22:35.868675 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.868637 2578 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 19:22:35.868712 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.868679 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 19:22:35.871551 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.871524 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Jul 06 19:22:35.916723 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.916603 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:22:35.917749 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.917720 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:22:35.917860 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.917758 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:22:35.917860 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.917779 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:22:35.917860 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.917815 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-94.ec2.internal" Jul 06 19:22:35.930659 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.930568 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-94.ec2.internal.18bfc9b4c9977e80\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c9977e80 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-94.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.775450752 +0000 UTC m=+0.404404712,LastTimestamp:2026-07-06 19:22:35.917740767 +0000 UTC m=+0.546694730,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:35.932040 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.932021 2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-140-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-140-94.ec2.internal" Jul 06 19:22:35.947993 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.947892 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-94.ec2.internal.18bfc9b4c997db07\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c997db07 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-94.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.775474439 +0000 UTC m=+0.404428399,LastTimestamp:2026-07-06 19:22:35.917762726 +0000 UTC m=+0.546716686,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:35.960542 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.960448 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-94.ec2.internal.18bfc9b4c99804fa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c99804fa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-94.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.775485178 +0000 UTC m=+0.404439139,LastTimestamp:2026-07-06 19:22:35.917785909 +0000 UTC m=+0.546739870,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:35.968855 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.968812 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal"] Jul 06 19:22:35.968981 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.968904 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:22:35.970951 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.970933 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:22:35.971070 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.970970 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:22:35.971070 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.970985 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:22:35.972241 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.972225 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:22:35.972367 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.972350 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal" Jul 06 19:22:35.972412 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.972392 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:22:35.975631 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.975610 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:22:35.975725 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.975650 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:22:35.975725 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.975665 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:22:35.975725 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.975618 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:22:35.975898 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.975741 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:22:35.975898 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.975757 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:22:35.975898 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.975884 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms" Jul 06 19:22:35.976485 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.975920 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-94.ec2.internal.18bfc9b4c9977e80\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c9977e80 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-94.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.775450752 +0000 UTC m=+0.404404712,LastTimestamp:2026-07-06 19:22:35.970951724 +0000 UTC m=+0.599905701,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:35.977209 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.977193 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal" Jul 06 19:22:35.977292 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.977226 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:22:35.978180 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.978161 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:22:35.978249 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.978198 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:22:35.978249 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:35.978212 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:22:35.997762 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:35.997655 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-94.ec2.internal.18bfc9b4c997db07\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c997db07 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-94.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.775474439 +0000 UTC m=+0.404428399,LastTimestamp:2026-07-06 19:22:35.970977053 +0000 UTC m=+0.599931013,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:36.008926 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.008897 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-94.ec2.internal\" not found" node="ip-10-0-140-94.ec2.internal" Jul 06 19:22:36.012269 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.012189 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-94.ec2.internal.18bfc9b4c99804fa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c99804fa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-94.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.775485178 +0000 UTC m=+0.404439139,LastTimestamp:2026-07-06 19:22:35.970990413 +0000 UTC m=+0.599944375,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:36.014064 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.014045 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-94.ec2.internal\" not found" node="ip-10-0-140-94.ec2.internal" Jul 06 19:22:36.028145 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.028058 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-94.ec2.internal.18bfc9b4c9977e80\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c9977e80 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-94.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.775450752 +0000 UTC m=+0.404404712,LastTimestamp:2026-07-06 19:22:35.97563646 +0000 UTC m=+0.604590429,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:36.044603 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.044511 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-94.ec2.internal.18bfc9b4c997db07\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c997db07 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-94.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.775474439 +0000 UTC m=+0.404428399,LastTimestamp:2026-07-06 19:22:35.975657105 +0000 UTC m=+0.604611072,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:36.056522 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.056494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf1609911ca9f346fada711322198d5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal\" (UID: \"cf1609911ca9f346fada711322198d5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal" Jul 06 19:22:36.056677 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.056527 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf1609911ca9f346fada711322198d5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal\" (UID: \"cf1609911ca9f346fada711322198d5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal" Jul 06 19:22:36.056677 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.056587 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aca81e81575313b78c60adfe6a1aa8fd-config\") pod \"kube-apiserver-proxy-ip-10-0-140-94.ec2.internal\" (UID: \"aca81e81575313b78c60adfe6a1aa8fd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal" Jul 06 19:22:36.058680 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.058600 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-94.ec2.internal.18bfc9b4c99804fa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c99804fa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-94.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.775485178 +0000 UTC m=+0.404439139,LastTimestamp:2026-07-06 19:22:35.975670524 +0000 UTC m=+0.604624488,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:36.069486 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.069395 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-94.ec2.internal.18bfc9b4c9977e80\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c9977e80 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-94.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.775450752 +0000 UTC m=+0.404404712,LastTimestamp:2026-07-06 19:22:35.975724139 +0000 UTC m=+0.604678108,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:36.081949 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.081859 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-94.ec2.internal.18bfc9b4c997db07\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c997db07 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-94.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.775474439 +0000 UTC m=+0.404428399,LastTimestamp:2026-07-06 19:22:35.975750524 +0000 UTC m=+0.604704486,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:36.093762 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.093733 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-67q7c" Jul 06 19:22:36.097007 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.096923 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-94.ec2.internal.18bfc9b4c99804fa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c99804fa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-94.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.775485178 +0000 UTC m=+0.404439139,LastTimestamp:2026-07-06 19:22:35.975761692 +0000 UTC m=+0.604715652,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:36.113497 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.113459 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-67q7c" Jul 06 19:22:36.113660 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.113583 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-94.ec2.internal.18bfc9b4c9977e80\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18bfc9b4c9977e80 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-94.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-07-06 19:22:35.775450752 +0000 UTC m=+0.404404712,LastTimestamp:2026-07-06 19:22:35.978180357 +0000 UTC m=+0.607134321,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}" Jul 06 19:22:36.132193 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.132160 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:22:36.133231 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.133211 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:22:36.133349 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.133249 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:22:36.133349 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.133264 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:22:36.133349 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.133303 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-94.ec2.internal" Jul 06 19:22:36.151899 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.151870 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-94.ec2.internal" Jul 06 19:22:36.152048 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.151904 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-94.ec2.internal\": node \"ip-10-0-140-94.ec2.internal\" not found" Jul 06 19:22:36.157056 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.157033 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf1609911ca9f346fada711322198d5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal\" (UID: \"cf1609911ca9f346fada711322198d5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal" Jul 06 19:22:36.157118 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.157068 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf1609911ca9f346fada711322198d5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal\" (UID: \"cf1609911ca9f346fada711322198d5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal" Jul 06 19:22:36.157118 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.157096 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aca81e81575313b78c60adfe6a1aa8fd-config\") pod \"kube-apiserver-proxy-ip-10-0-140-94.ec2.internal\" (UID: \"aca81e81575313b78c60adfe6a1aa8fd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal" Jul 06 19:22:36.157188 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.157147 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf1609911ca9f346fada711322198d5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal\" (UID: \"cf1609911ca9f346fada711322198d5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal" Jul 06 19:22:36.157188 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.157162 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aca81e81575313b78c60adfe6a1aa8fd-config\") pod \"kube-apiserver-proxy-ip-10-0-140-94.ec2.internal\" (UID: \"aca81e81575313b78c60adfe6a1aa8fd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal" Jul 06 19:22:36.157188 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.157144 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf1609911ca9f346fada711322198d5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal\" (UID: \"cf1609911ca9f346fada711322198d5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal" Jul 06 19:22:36.183083 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.182995 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found" Jul 06 19:22:36.285256 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.285202 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found" Jul 06 19:22:36.313520 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.313483 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal" Jul 06 19:22:36.317855 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.317607 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal" Jul 06 19:22:36.385565 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.385519 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found" Jul 06 19:22:36.486743 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.486626 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found" Jul 06 19:22:36.587097 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.587062 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found" Jul 06 19:22:36.597238 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.597203 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:22:36.689167 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.689113 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 19:22:36.689167 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.689118 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found" Jul 06 19:22:36.689658 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.689308 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:22:36.789350 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.789249 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found" Jul 06 19:22:36.889647 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.889603 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found" Jul 06 19:22:36.963414 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.963383 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:22:36.964927 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.964904 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:22:36.965083 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.964943 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:22:36.965083 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.964961 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:22:36.979743 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:36.979710 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca81e81575313b78c60adfe6a1aa8fd.slice/crio-0a0a1b51294364095e62a0535e3a548e0d1be093672d39a86d5f971513e24bc6 WatchSource:0}: Error finding container 0a0a1b51294364095e62a0535e3a548e0d1be093672d39a86d5f971513e24bc6: Status 404 returned error can't find the container with id 0a0a1b51294364095e62a0535e3a548e0d1be093672d39a86d5f971513e24bc6 Jul 06 19:22:36.979743 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.979734 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:22:36.980452 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:36.980429 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf1609911ca9f346fada711322198d5a.slice/crio-c6b5023c3adb42a3855e6c27107758738effb8a29cf3beb1df4bbdbc68be2af3 WatchSource:0}: Error finding container c6b5023c3adb42a3855e6c27107758738effb8a29cf3beb1df4bbdbc68be2af3: Status 404 returned error can't find the container with id c6b5023c3adb42a3855e6c27107758738effb8a29cf3beb1df4bbdbc68be2af3 Jul 06 19:22:36.985426 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:36.985410 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:22:36.990268 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:36.990245 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found" Jul 06 19:22:37.092743 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:37.092615 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found" Jul 06 19:22:37.117580 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.117502 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 19:17:36 +0000 UTC" deadline="2027-12-01 22:37:31.364666141 +0000 UTC" Jul 06 19:22:37.117580 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.117579 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12315h14m54.247091567s" Jul 06 19:22:37.194710 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:37.194667 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found" Jul 06 19:22:37.209523 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.209492 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:22:37.254892 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.254850 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal" Jul 06 19:22:37.278455 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.278430 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:22:37.279340 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.279324 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal" Jul 06 19:22:37.296998 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.296973 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:22:37.457608 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.457515 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:22:37.738184 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.738072 2578 apiserver.go:52] "Watching apiserver" Jul 06 19:22:37.746620 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.746581 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 19:22:37.746952 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.746926 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-kpsb4","kube-system/konnectivity-agent-wcsxd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal","openshift-multus/network-metrics-daemon-lh7hd","openshift-network-diagnostics/network-check-target-vfp2h","openshift-network-operator/iptables-alerter-nx2gf","openshift-ovn-kubernetes/ovnkube-node-lqwfz","kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal","openshift-cluster-node-tuning-operator/tuned-pg5z7","openshift-image-registry/node-ca-qfm4f","openshift-multus/multus-additional-cni-plugins-wk4zk","openshift-multus/multus-c2cs8"] Jul 06 19:22:37.749019 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.749004 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.750022 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.750002 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wcsxd" Jul 06 19:22:37.751184 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.751161 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.752296 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.752274 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:37.752387 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:37.752341 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lh7hd" podUID="362f1775-4ab7-400a-b216-a22e4ca6c9fb" Jul 06 19:22:37.752682 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.752666 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 19:22:37.752922 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.752904 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-76tlm\"" Jul 06 19:22:37.753208 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.753188 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:22:37.753451 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.753431 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:22:37.753519 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:37.753499 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfp2h" podUID="abb3d52d-bb53-47f0-9708-075b2eae0e2d" Jul 06 19:22:37.753887 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.753872 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5f8zd\"" Jul 06 19:22:37.754017 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.753998 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 19:22:37.754566 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.754547 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nx2gf" Jul 06 19:22:37.755910 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.755891 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 19:22:37.756011 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.755962 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.757321 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.757227 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 19:22:37.757321 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.757268 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 19:22:37.757321 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.757290 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 19:22:37.757579 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.757542 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:22:37.757648 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.757605 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 19:22:37.757752 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.757729 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qfm4f" Jul 06 19:22:37.757856 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.757834 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l7t67\"" Jul 06 19:22:37.758439 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.758418 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 19:22:37.759003 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.758985 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.759835 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.759813 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 19:22:37.760038 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.760020 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 19:22:37.760038 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.760040 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 19:22:37.760171 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.760135 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dl52g\"" Jul 06 19:22:37.760358 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.760339 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 19:22:37.760463 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.760445 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.760463 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.760457 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 19:22:37.760572 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.760497 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 19:22:37.760572 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.760567 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 19:22:37.760689 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.760671 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vht6l\"" Jul 06 19:22:37.761175 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.761159 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pkb6c\"" Jul 06 19:22:37.761605 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.761590 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 19:22:37.761821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.761804 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:37.761884 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:37.761859 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpsb4" podUID="a6d9fe54-0654-4f9d-bb79-0fbeec14c98f" Jul 06 19:22:37.763141 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.763125 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 19:22:37.763192 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.763153 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 19:22:37.763290 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.763264 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 19:22:37.766631 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.766534 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.766631 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.766586 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-env-overrides\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.766631 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.766625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-ovn-node-metrics-cert\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.766880 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.766794 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-kubernetes\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.766880 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.766830 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-sysctl-conf\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.766880 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.766864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/140ec7e0-97cd-45df-b179-e62e429073b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.767029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.766898 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-ovnkube-config\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.767029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.766927 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/63219616-9c15-4abc-90ee-42160928e479-konnectivity-ca\") pod \"konnectivity-agent-wcsxd\" (UID: \"63219616-9c15-4abc-90ee-42160928e479\") " pod="kube-system/konnectivity-agent-wcsxd" Jul 06 19:22:37.767029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.766961 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a56948c7-e923-402c-a264-1ac5dec90393-host-slash\") pod \"iptables-alerter-nx2gf\" (UID: \"a56948c7-e923-402c-a264-1ac5dec90393\") " pod="openshift-network-operator/iptables-alerter-nx2gf" Jul 06 19:22:37.767029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.766997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-kubelet\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.767224 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767028 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/363d0018-46e0-4f8c-bf7c-3c582c493c88-tmp\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.767224 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767070 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/140ec7e0-97cd-45df-b179-e62e429073b4-registration-dir\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.767224 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767097 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/140ec7e0-97cd-45df-b179-e62e429073b4-device-dir\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.767224 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767131 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-run-ovn-kubernetes\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.767224 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767163 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-cni-netd\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.767224 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767200 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/63219616-9c15-4abc-90ee-42160928e479-agent-certs\") pod \"konnectivity-agent-wcsxd\" (UID: \"63219616-9c15-4abc-90ee-42160928e479\") " pod="kube-system/konnectivity-agent-wcsxd" Jul 06 19:22:37.767539 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767230 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs\") pod \"network-metrics-daemon-lh7hd\" (UID: \"362f1775-4ab7-400a-b216-a22e4ca6c9fb\") " pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:37.767539 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a56948c7-e923-402c-a264-1ac5dec90393-iptables-alerter-script\") pod \"iptables-alerter-nx2gf\" (UID: \"a56948c7-e923-402c-a264-1ac5dec90393\") " pod="openshift-network-operator/iptables-alerter-nx2gf" Jul 06 19:22:37.767539 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-ovnkube-script-lib\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.767539 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-lib-modules\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.767539 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767467 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pp42\" (UniqueName: \"kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42\") pod \"network-check-target-vfp2h\" (UID: \"abb3d52d-bb53-47f0-9708-075b2eae0e2d\") " pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:22:37.767539 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767518 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2zb7\" (UniqueName: \"kubernetes.io/projected/a56948c7-e923-402c-a264-1ac5dec90393-kube-api-access-m2zb7\") pod \"iptables-alerter-nx2gf\" (UID: \"a56948c7-e923-402c-a264-1ac5dec90393\") " pod="openshift-network-operator/iptables-alerter-nx2gf" Jul 06 19:22:37.767827 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767567 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-run-netns\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.767827 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767600 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jp8n\" (UniqueName: \"kubernetes.io/projected/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-kube-api-access-5jp8n\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.767827 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767651 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-sysctl-d\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.767827 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767684 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-var-lib-kubelet\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.767827 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767712 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/140ec7e0-97cd-45df-b179-e62e429073b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.767827 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-host\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.767827 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767821 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-slash\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.768136 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767854 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-run-openvswitch\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.768136 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767888 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-log-socket\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.768136 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767914 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-systemd\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.768136 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767946 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/140ec7e0-97cd-45df-b179-e62e429073b4-sys-fs\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.768136 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.767980 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zm2q\" (UniqueName: \"kubernetes.io/projected/140ec7e0-97cd-45df-b179-e62e429073b4-kube-api-access-9zm2q\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.768136 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.768016 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-run-ovn\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.768136 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.768105 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6c5b\" (UniqueName: \"kubernetes.io/projected/362f1775-4ab7-400a-b216-a22e4ca6c9fb-kube-api-access-p6c5b\") pod \"network-metrics-daemon-lh7hd\" (UID: \"362f1775-4ab7-400a-b216-a22e4ca6c9fb\") " pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:37.768451 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.768140 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-systemd-units\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.768451 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.768174 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-var-lib-openvswitch\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.768451 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.768211 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-etc-openvswitch\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.768451 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.768242 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-run\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.768451 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.768273 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-sys\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.768451 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.768305 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-node-log\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.768451 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.768331 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-modprobe-d\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.768451 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.768366 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-sysconfig\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.768451 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.768399 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-tuned\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.768451 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.768431 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fhbt\" (UniqueName: \"kubernetes.io/projected/363d0018-46e0-4f8c-bf7c-3c582c493c88-kube-api-access-7fhbt\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.768985 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.768467 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/140ec7e0-97cd-45df-b179-e62e429073b4-socket-dir\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.768985 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.768507 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-run-systemd\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.768985 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.768543 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-cni-bin\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.769466 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.769302 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 19:22:37.769573 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.769557 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 19:22:37.769809 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.769787 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-92rgc\"" Jul 06 19:22:37.770190 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.770169 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 19:22:37.771049 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.771032 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cf67r\"" Jul 06 19:22:37.775672 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.775648 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 19:22:37.776351 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.776337 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 19:22:37.780708 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.780687 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:22:37.848861 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.848817 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4zs96" Jul 06 19:22:37.855261 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.855231 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 19:22:37.861287 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.861254 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4zs96" Jul 06 19:22:37.869564 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.869518 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e4c4a93-9461-4a11-aef0-e1399f9bc611-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.869564 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.869562 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-cnibin\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.869821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.869597 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-slash\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.869821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.869626 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/140ec7e0-97cd-45df-b179-e62e429073b4-sys-fs\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.869821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.869655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zm2q\" (UniqueName: \"kubernetes.io/projected/140ec7e0-97cd-45df-b179-e62e429073b4-kube-api-access-9zm2q\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.869821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.869677 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-multus-conf-dir\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.869821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.869695 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-kubelet-config\") pod \"global-pull-secret-syncer-kpsb4\" (UID: \"a6d9fe54-0654-4f9d-bb79-0fbeec14c98f\") " pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:37.869821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.869720 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6c5b\" (UniqueName: \"kubernetes.io/projected/362f1775-4ab7-400a-b216-a22e4ca6c9fb-kube-api-access-p6c5b\") pod \"network-metrics-daemon-lh7hd\" (UID: \"362f1775-4ab7-400a-b216-a22e4ca6c9fb\") " pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:37.869821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.869743 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-var-lib-openvswitch\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.869821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.869758 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-etc-openvswitch\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.869821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.869796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-tuned\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.869821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.869814 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fhbt\" (UniqueName: \"kubernetes.io/projected/363d0018-46e0-4f8c-bf7c-3c582c493c88-kube-api-access-7fhbt\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.870339 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.869830 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e4c4a93-9461-4a11-aef0-e1399f9bc611-cni-binary-copy\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.870339 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.869848 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-host-var-lib-kubelet\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.870339 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.869868 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/140ec7e0-97cd-45df-b179-e62e429073b4-socket-dir\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.870339 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.869896 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-host-var-lib-cni-bin\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.870339 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.869942 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-dbus\") pod \"global-pull-secret-syncer-kpsb4\" (UID: \"a6d9fe54-0654-4f9d-bb79-0fbeec14c98f\") " pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:37.870339 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.869967 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnw8p\" (UniqueName: \"kubernetes.io/projected/ecbc6515-e4c1-489d-ac42-dd3a967b782c-kube-api-access-lnw8p\") pod \"node-ca-qfm4f\" (UID: \"ecbc6515-e4c1-489d-ac42-dd3a967b782c\") " pod="openshift-image-registry/node-ca-qfm4f" Jul 06 19:22:37.870339 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.869995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-env-overrides\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.870339 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870020 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/140ec7e0-97cd-45df-b179-e62e429073b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.870339 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870045 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-cni-binary-copy\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.870339 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870075 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-etc-kubernetes\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.870339 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870097 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ecbc6515-e4c1-489d-ac42-dd3a967b782c-host\") pod \"node-ca-qfm4f\" (UID: \"ecbc6515-e4c1-489d-ac42-dd3a967b782c\") " pod="openshift-image-registry/node-ca-qfm4f" Jul 06 19:22:37.870339 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870198 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-slash\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.870821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870376 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/140ec7e0-97cd-45df-b179-e62e429073b4-socket-dir\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.870821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870420 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e4c4a93-9461-4a11-aef0-e1399f9bc611-system-cni-dir\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.870821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870439 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-var-lib-openvswitch\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.870821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870461 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/140ec7e0-97cd-45df-b179-e62e429073b4-sys-fs\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.870821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/63219616-9c15-4abc-90ee-42160928e479-konnectivity-ca\") pod \"konnectivity-agent-wcsxd\" (UID: \"63219616-9c15-4abc-90ee-42160928e479\") " pod="kube-system/konnectivity-agent-wcsxd" Jul 06 19:22:37.870821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870513 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 19:22:37.870821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a56948c7-e923-402c-a264-1ac5dec90393-host-slash\") pod \"iptables-alerter-nx2gf\" (UID: \"a56948c7-e923-402c-a264-1ac5dec90393\") " pod="openshift-network-operator/iptables-alerter-nx2gf" Jul 06 19:22:37.870821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870575 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-etc-openvswitch\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.870821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/363d0018-46e0-4f8c-bf7c-3c582c493c88-tmp\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.870821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870610 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj4nj\" (UniqueName: \"kubernetes.io/projected/3e4c4a93-9461-4a11-aef0-e1399f9bc611-kube-api-access-rj4nj\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.870821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870616 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/140ec7e0-97cd-45df-b179-e62e429073b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.870821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870635 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-run-ovn-kubernetes\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.870821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870659 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-lib-modules\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.870821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870789 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-lib-modules\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.871420 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.870843 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-run-ovn-kubernetes\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.871420 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871030 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a56948c7-e923-402c-a264-1ac5dec90393-host-slash\") pod \"iptables-alerter-nx2gf\" (UID: \"a56948c7-e923-402c-a264-1ac5dec90393\") " pod="openshift-network-operator/iptables-alerter-nx2gf" Jul 06 19:22:37.871420 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871056 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-env-overrides\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.871420 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871056 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/63219616-9c15-4abc-90ee-42160928e479-konnectivity-ca\") pod \"konnectivity-agent-wcsxd\" (UID: \"63219616-9c15-4abc-90ee-42160928e479\") " pod="kube-system/konnectivity-agent-wcsxd" Jul 06 19:22:37.871420 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871171 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pp42\" (UniqueName: \"kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42\") pod \"network-check-target-vfp2h\" (UID: \"abb3d52d-bb53-47f0-9708-075b2eae0e2d\") " pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:22:37.871420 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871207 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-system-cni-dir\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.871420 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871232 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-os-release\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.871420 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871259 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-multus-daemon-config\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.871420 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871288 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2zb7\" (UniqueName: \"kubernetes.io/projected/a56948c7-e923-402c-a264-1ac5dec90393-kube-api-access-m2zb7\") pod \"iptables-alerter-nx2gf\" (UID: \"a56948c7-e923-402c-a264-1ac5dec90393\") " pod="openshift-network-operator/iptables-alerter-nx2gf" Jul 06 19:22:37.871420 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-run-netns\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.871420 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871340 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jp8n\" (UniqueName: \"kubernetes.io/projected/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-kube-api-access-5jp8n\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.871420 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871370 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e4c4a93-9461-4a11-aef0-e1399f9bc611-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.871420 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871399 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-multus-cni-dir\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.871420 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871423 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-host\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.872062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871435 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-run-netns\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.872062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871447 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-hostroot\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.872062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871476 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-run-openvswitch\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.872062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871501 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-log-socket\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.872062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871527 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-systemd\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.872062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-run-ovn\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.872062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871597 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-systemd-units\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.872062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-run\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.872062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871648 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-sys\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.872062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871674 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-host-run-netns\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.872062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-node-log\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.872062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871732 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-modprobe-d\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.872062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871756 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-sysconfig\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.872062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871766 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-run-ovn\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.872062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871797 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-host-run-multus-certs\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.872062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871814 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-sys\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.872062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-run-systemd\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.872062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871861 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-systemd-units\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.873029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871865 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-systemd\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.873029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871598 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-host\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.873029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871872 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-cni-bin\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.873029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871905 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-log-socket\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.873029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871918 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.873029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871946 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-run-systemd\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.873029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-ovn-node-metrics-cert\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.873029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871962 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-sysconfig\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.873029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871979 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-kubernetes\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.873029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871995 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-node-log\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.873029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.871817 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-run-openvswitch\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.873029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872006 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-sysctl-conf\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.873029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872037 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-cni-bin\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.873029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872036 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-multus-socket-dir-parent\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.873029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872057 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-modprobe-d\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.873029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872083 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.873029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872084 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e4c4a93-9461-4a11-aef0-e1399f9bc611-cnibin\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.873961 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872105 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-sysctl-conf\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.873961 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872119 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-run\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.873961 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872122 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3e4c4a93-9461-4a11-aef0-e1399f9bc611-os-release\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.873961 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-ovnkube-config\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.873961 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872160 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-kubernetes\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.873961 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-kubelet\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.873961 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872204 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/140ec7e0-97cd-45df-b179-e62e429073b4-registration-dir\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.873961 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872229 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/140ec7e0-97cd-45df-b179-e62e429073b4-device-dir\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.873961 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872257 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e4c4a93-9461-4a11-aef0-e1399f9bc611-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.873961 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872284 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fln9q\" (UniqueName: \"kubernetes.io/projected/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-kube-api-access-fln9q\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.873961 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872298 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/140ec7e0-97cd-45df-b179-e62e429073b4-registration-dir\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.873961 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-cni-netd\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.873961 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872257 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-kubelet\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.873961 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872354 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-host-cni-netd\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.873961 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872361 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/140ec7e0-97cd-45df-b179-e62e429073b4-device-dir\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.873961 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872381 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/63219616-9c15-4abc-90ee-42160928e479-agent-certs\") pod \"konnectivity-agent-wcsxd\" (UID: \"63219616-9c15-4abc-90ee-42160928e479\") " pod="kube-system/konnectivity-agent-wcsxd" Jul 06 19:22:37.873961 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872406 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs\") pod \"network-metrics-daemon-lh7hd\" (UID: \"362f1775-4ab7-400a-b216-a22e4ca6c9fb\") " pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:37.874883 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872437 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a56948c7-e923-402c-a264-1ac5dec90393-iptables-alerter-script\") pod \"iptables-alerter-nx2gf\" (UID: \"a56948c7-e923-402c-a264-1ac5dec90393\") " pod="openshift-network-operator/iptables-alerter-nx2gf" Jul 06 19:22:37.874883 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:37.872605 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:22:37.874883 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:37.872686 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs podName:362f1775-4ab7-400a-b216-a22e4ca6c9fb nodeName:}" failed. No retries permitted until 2026-07-06 19:22:38.372660742 +0000 UTC m=+3.001614708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs") pod "network-metrics-daemon-lh7hd" (UID: "362f1775-4ab7-400a-b216-a22e4ca6c9fb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:22:37.874883 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872887 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-ovnkube-script-lib\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.874883 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872923 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-host-var-lib-cni-multus\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.874883 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872956 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret\") pod \"global-pull-secret-syncer-kpsb4\" (UID: \"a6d9fe54-0654-4f9d-bb79-0fbeec14c98f\") " pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:37.874883 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.872997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ecbc6515-e4c1-489d-ac42-dd3a967b782c-serviceca\") pod \"node-ca-qfm4f\" (UID: \"ecbc6515-e4c1-489d-ac42-dd3a967b782c\") " pod="openshift-image-registry/node-ca-qfm4f" Jul 06 19:22:37.874883 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.873011 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a56948c7-e923-402c-a264-1ac5dec90393-iptables-alerter-script\") pod \"iptables-alerter-nx2gf\" (UID: \"a56948c7-e923-402c-a264-1ac5dec90393\") " pod="openshift-network-operator/iptables-alerter-nx2gf" Jul 06 19:22:37.874883 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.873026 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-sysctl-d\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.874883 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.873054 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-var-lib-kubelet\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.874883 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.873080 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/140ec7e0-97cd-45df-b179-e62e429073b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.874883 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.873109 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-host-run-k8s-cni-cncf-io\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.874883 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.873190 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-var-lib-kubelet\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.874883 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.873220 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/140ec7e0-97cd-45df-b179-e62e429073b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.874883 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.873233 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-sysctl-d\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.874883 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.873644 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-ovnkube-config\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.874883 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.874024 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-ovnkube-script-lib\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.875639 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.875028 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/363d0018-46e0-4f8c-bf7c-3c582c493c88-tmp\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.875639 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.875375 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/363d0018-46e0-4f8c-bf7c-3c582c493c88-etc-tuned\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.875639 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.875564 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-ovn-node-metrics-cert\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.876235 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.876185 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal" event={"ID":"aca81e81575313b78c60adfe6a1aa8fd","Type":"ContainerStarted","Data":"0a0a1b51294364095e62a0535e3a548e0d1be093672d39a86d5f971513e24bc6"} Jul 06 19:22:37.877303 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.877176 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/63219616-9c15-4abc-90ee-42160928e479-agent-certs\") pod \"konnectivity-agent-wcsxd\" (UID: \"63219616-9c15-4abc-90ee-42160928e479\") " pod="kube-system/konnectivity-agent-wcsxd" Jul 06 19:22:37.878237 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.878211 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal" event={"ID":"cf1609911ca9f346fada711322198d5a","Type":"ContainerStarted","Data":"c6b5023c3adb42a3855e6c27107758738effb8a29cf3beb1df4bbdbc68be2af3"} Jul 06 19:22:37.893503 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:37.893472 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:22:37.893503 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:37.893506 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:22:37.893746 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:37.893521 2578 projected.go:196] Error preparing data for projected volume kube-api-access-8pp42 for pod openshift-network-diagnostics/network-check-target-vfp2h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:22:37.893746 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:37.893610 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42 podName:abb3d52d-bb53-47f0-9708-075b2eae0e2d nodeName:}" failed. No retries permitted until 2026-07-06 19:22:38.393589087 +0000 UTC m=+3.022543050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8pp42" (UniqueName: "kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42") pod "network-check-target-vfp2h" (UID: "abb3d52d-bb53-47f0-9708-075b2eae0e2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:22:37.894220 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.894186 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2zb7\" (UniqueName: \"kubernetes.io/projected/a56948c7-e923-402c-a264-1ac5dec90393-kube-api-access-m2zb7\") pod \"iptables-alerter-nx2gf\" (UID: \"a56948c7-e923-402c-a264-1ac5dec90393\") " pod="openshift-network-operator/iptables-alerter-nx2gf" Jul 06 19:22:37.896208 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.896180 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zm2q\" (UniqueName: \"kubernetes.io/projected/140ec7e0-97cd-45df-b179-e62e429073b4-kube-api-access-9zm2q\") pod \"aws-ebs-csi-driver-node-g9qhn\" (UID: \"140ec7e0-97cd-45df-b179-e62e429073b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:37.898244 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.898214 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jp8n\" (UniqueName: \"kubernetes.io/projected/0b2f8393-722b-4d31-b5ad-702a91f1ebc8-kube-api-access-5jp8n\") pod \"ovnkube-node-lqwfz\" (UID: \"0b2f8393-722b-4d31-b5ad-702a91f1ebc8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:37.899468 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.899443 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6c5b\" (UniqueName: \"kubernetes.io/projected/362f1775-4ab7-400a-b216-a22e4ca6c9fb-kube-api-access-p6c5b\") pod \"network-metrics-daemon-lh7hd\" (UID: \"362f1775-4ab7-400a-b216-a22e4ca6c9fb\") " pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:37.899796 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.899757 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fhbt\" (UniqueName: \"kubernetes.io/projected/363d0018-46e0-4f8c-bf7c-3c582c493c88-kube-api-access-7fhbt\") pod \"tuned-pg5z7\" (UID: \"363d0018-46e0-4f8c-bf7c-3c582c493c88\") " pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:37.974483 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.974434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-host-run-k8s-cni-cncf-io\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.974679 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.974493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e4c4a93-9461-4a11-aef0-e1399f9bc611-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.974679 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.974528 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-cnibin\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.974679 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.974558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-multus-conf-dir\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.974679 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.974588 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-kubelet-config\") pod \"global-pull-secret-syncer-kpsb4\" (UID: \"a6d9fe54-0654-4f9d-bb79-0fbeec14c98f\") " pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:37.974679 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.974591 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-host-run-k8s-cni-cncf-io\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.974679 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.974619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e4c4a93-9461-4a11-aef0-e1399f9bc611-cni-binary-copy\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.975007 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.974747 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-cnibin\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.975007 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.974791 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-host-var-lib-kubelet\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.975007 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.974822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-host-var-lib-cni-bin\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.975007 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.974851 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-dbus\") pod \"global-pull-secret-syncer-kpsb4\" (UID: \"a6d9fe54-0654-4f9d-bb79-0fbeec14c98f\") " pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:37.975007 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.974880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnw8p\" (UniqueName: \"kubernetes.io/projected/ecbc6515-e4c1-489d-ac42-dd3a967b782c-kube-api-access-lnw8p\") pod \"node-ca-qfm4f\" (UID: \"ecbc6515-e4c1-489d-ac42-dd3a967b782c\") " pod="openshift-image-registry/node-ca-qfm4f" Jul 06 19:22:37.975007 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.974896 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-multus-conf-dir\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.975007 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.974912 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-cni-binary-copy\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.975007 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.974954 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-etc-kubernetes\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.975007 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.974982 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ecbc6515-e4c1-489d-ac42-dd3a967b782c-host\") pod \"node-ca-qfm4f\" (UID: \"ecbc6515-e4c1-489d-ac42-dd3a967b782c\") " pod="openshift-image-registry/node-ca-qfm4f" Jul 06 19:22:37.975007 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e4c4a93-9461-4a11-aef0-e1399f9bc611-system-cni-dir\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.975522 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975038 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj4nj\" (UniqueName: \"kubernetes.io/projected/3e4c4a93-9461-4a11-aef0-e1399f9bc611-kube-api-access-rj4nj\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.975522 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975090 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-system-cni-dir\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.975522 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975114 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-os-release\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.975522 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975138 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-multus-daemon-config\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.975522 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e4c4a93-9461-4a11-aef0-e1399f9bc611-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.975522 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975193 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-multus-cni-dir\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.975522 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975216 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-hostroot\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.975522 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975250 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-host-run-netns\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.975522 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-host-run-multus-certs\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.975522 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-multus-socket-dir-parent\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.975522 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975308 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e4c4a93-9461-4a11-aef0-e1399f9bc611-cni-binary-copy\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.975522 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e4c4a93-9461-4a11-aef0-e1399f9bc611-cnibin\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.975522 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975356 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3e4c4a93-9461-4a11-aef0-e1399f9bc611-os-release\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.975522 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975362 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e4c4a93-9461-4a11-aef0-e1399f9bc611-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.975522 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e4c4a93-9461-4a11-aef0-e1399f9bc611-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.975522 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975389 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-kubelet-config\") pod \"global-pull-secret-syncer-kpsb4\" (UID: \"a6d9fe54-0654-4f9d-bb79-0fbeec14c98f\") " pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:37.975522 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975437 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fln9q\" (UniqueName: \"kubernetes.io/projected/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-kube-api-access-fln9q\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.976437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975451 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-cni-binary-copy\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.976437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975452 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ecbc6515-e4c1-489d-ac42-dd3a967b782c-host\") pod \"node-ca-qfm4f\" (UID: \"ecbc6515-e4c1-489d-ac42-dd3a967b782c\") " pod="openshift-image-registry/node-ca-qfm4f" Jul 06 19:22:37.976437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975488 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e4c4a93-9461-4a11-aef0-e1399f9bc611-system-cni-dir\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.976437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975453 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-host-run-netns\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.976437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975509 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-host-run-multus-certs\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.976437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-host-var-lib-cni-multus\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.976437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975542 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3e4c4a93-9461-4a11-aef0-e1399f9bc611-os-release\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.976437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975557 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-os-release\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.976437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975572 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e4c4a93-9461-4a11-aef0-e1399f9bc611-cnibin\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.976437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975617 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-multus-socket-dir-parent\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.976437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975633 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e4c4a93-9461-4a11-aef0-e1399f9bc611-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.976437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975669 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-multus-cni-dir\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.976437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975678 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-hostroot\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.976437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975745 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-host-var-lib-cni-multus\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.976437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975803 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-dbus\") pod \"global-pull-secret-syncer-kpsb4\" (UID: \"a6d9fe54-0654-4f9d-bb79-0fbeec14c98f\") " pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:37.976437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975832 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-system-cni-dir\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.976437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975845 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-host-var-lib-kubelet\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.976437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.975893 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-host-var-lib-cni-bin\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.977295 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.976070 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret\") pod \"global-pull-secret-syncer-kpsb4\" (UID: \"a6d9fe54-0654-4f9d-bb79-0fbeec14c98f\") " pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:37.977295 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.976073 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-multus-daemon-config\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.977295 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.976119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ecbc6515-e4c1-489d-ac42-dd3a967b782c-serviceca\") pod \"node-ca-qfm4f\" (UID: \"ecbc6515-e4c1-489d-ac42-dd3a967b782c\") " pod="openshift-image-registry/node-ca-qfm4f" Jul 06 19:22:37.977295 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:37.976167 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:22:37.977295 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.976174 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e4c4a93-9461-4a11-aef0-e1399f9bc611-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:37.977295 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.976214 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-etc-kubernetes\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:37.977295 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:37.976229 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret podName:a6d9fe54-0654-4f9d-bb79-0fbeec14c98f nodeName:}" failed. No retries permitted until 2026-07-06 19:22:38.476209877 +0000 UTC m=+3.105163836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret") pod "global-pull-secret-syncer-kpsb4" (UID: "a6d9fe54-0654-4f9d-bb79-0fbeec14c98f") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:22:37.977295 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.976605 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ecbc6515-e4c1-489d-ac42-dd3a967b782c-serviceca\") pod \"node-ca-qfm4f\" (UID: \"ecbc6515-e4c1-489d-ac42-dd3a967b782c\") " pod="openshift-image-registry/node-ca-qfm4f" Jul 06 19:22:37.988810 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.988675 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnw8p\" (UniqueName: \"kubernetes.io/projected/ecbc6515-e4c1-489d-ac42-dd3a967b782c-kube-api-access-lnw8p\") pod \"node-ca-qfm4f\" (UID: \"ecbc6515-e4c1-489d-ac42-dd3a967b782c\") " pod="openshift-image-registry/node-ca-qfm4f" Jul 06 19:22:37.998733 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:37.998688 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fln9q\" (UniqueName: \"kubernetes.io/projected/1be1a75c-b7c1-466c-99cb-bdc2e3749d87-kube-api-access-fln9q\") pod \"multus-c2cs8\" (UID: \"1be1a75c-b7c1-466c-99cb-bdc2e3749d87\") " pod="openshift-multus/multus-c2cs8" Jul 06 19:22:38.001072 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.001010 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj4nj\" (UniqueName: \"kubernetes.io/projected/3e4c4a93-9461-4a11-aef0-e1399f9bc611-kube-api-access-rj4nj\") pod \"multus-additional-cni-plugins-wk4zk\" (UID: \"3e4c4a93-9461-4a11-aef0-e1399f9bc611\") " pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:38.060432 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.060386 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" Jul 06 19:22:38.066444 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.066419 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wcsxd" Jul 06 19:22:38.071226 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:38.071192 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363d0018_46e0_4f8c_bf7c_3c582c493c88.slice/crio-64a369813a404d297591a81684ba578bda6814430b15061883df61d34cb936c0 WatchSource:0}: Error finding container 64a369813a404d297591a81684ba578bda6814430b15061883df61d34cb936c0: Status 404 returned error can't find the container with id 64a369813a404d297591a81684ba578bda6814430b15061883df61d34cb936c0 Jul 06 19:22:38.079147 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.079123 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" Jul 06 19:22:38.079646 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:38.079595 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63219616_9c15_4abc_90ee_42160928e479.slice/crio-8b985f8c8ebaa79911ab19e313a90bfcd1e0fc7b945d5e71f7ec3a48b2cfec2e WatchSource:0}: Error finding container 8b985f8c8ebaa79911ab19e313a90bfcd1e0fc7b945d5e71f7ec3a48b2cfec2e: Status 404 returned error can't find the container with id 8b985f8c8ebaa79911ab19e313a90bfcd1e0fc7b945d5e71f7ec3a48b2cfec2e Jul 06 19:22:38.082999 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.082976 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nx2gf" Jul 06 19:22:38.088585 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.088562 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:22:38.090240 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:38.090211 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140ec7e0_97cd_45df_b179_e62e429073b4.slice/crio-05eecbebbc6cb1287bfbcf4c1f1959e5af023e94df426dd9314b629773a64a2b WatchSource:0}: Error finding container 05eecbebbc6cb1287bfbcf4c1f1959e5af023e94df426dd9314b629773a64a2b: Status 404 returned error can't find the container with id 05eecbebbc6cb1287bfbcf4c1f1959e5af023e94df426dd9314b629773a64a2b Jul 06 19:22:38.093535 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:38.093420 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda56948c7_e923_402c_a264_1ac5dec90393.slice/crio-ba67606c7030eb60e3d69871d640e31fc16397ddb466c7fa866f5717bf2f67fe WatchSource:0}: Error finding container ba67606c7030eb60e3d69871d640e31fc16397ddb466c7fa866f5717bf2f67fe: Status 404 returned error can't find the container with id ba67606c7030eb60e3d69871d640e31fc16397ddb466c7fa866f5717bf2f67fe Jul 06 19:22:38.094564 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.094381 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qfm4f" Jul 06 19:22:38.100180 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:38.100145 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2f8393_722b_4d31_b5ad_702a91f1ebc8.slice/crio-e1096b527b4129772ad49dcd6f598ee4d8e5491d4f941c90b6bbf5493923b44a WatchSource:0}: Error finding container e1096b527b4129772ad49dcd6f598ee4d8e5491d4f941c90b6bbf5493923b44a: Status 404 returned error can't find the container with id e1096b527b4129772ad49dcd6f598ee4d8e5491d4f941c90b6bbf5493923b44a Jul 06 19:22:38.101410 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.101093 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wk4zk" Jul 06 19:22:38.105268 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.105248 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c2cs8" Jul 06 19:22:38.107916 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:38.107892 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbc6515_e4c1_489d_ac42_dd3a967b782c.slice/crio-64450ac91ef85dea93a288f13a233dda89f2829e736984afbd95eb47d05e0bde WatchSource:0}: Error finding container 64450ac91ef85dea93a288f13a233dda89f2829e736984afbd95eb47d05e0bde: Status 404 returned error can't find the container with id 64450ac91ef85dea93a288f13a233dda89f2829e736984afbd95eb47d05e0bde Jul 06 19:22:38.114121 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:38.114065 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4c4a93_9461_4a11_aef0_e1399f9bc611.slice/crio-4b8393c61ec7fa0b7700c54137886c9cb9a734831c2b660b3d2d9c3aa14af30d WatchSource:0}: Error finding container 4b8393c61ec7fa0b7700c54137886c9cb9a734831c2b660b3d2d9c3aa14af30d: Status 404 returned error can't find the container with id 4b8393c61ec7fa0b7700c54137886c9cb9a734831c2b660b3d2d9c3aa14af30d Jul 06 19:22:38.118175 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:22:38.117690 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be1a75c_b7c1_466c_99cb_bdc2e3749d87.slice/crio-337f10fe505aad09bccc473929759d6bea86f8a3e2241f5ed14ef919d8467b1b WatchSource:0}: Error finding container 337f10fe505aad09bccc473929759d6bea86f8a3e2241f5ed14ef919d8467b1b: Status 404 returned error can't find the container with id 337f10fe505aad09bccc473929759d6bea86f8a3e2241f5ed14ef919d8467b1b Jul 06 19:22:38.380363 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.380270 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs\") pod \"network-metrics-daemon-lh7hd\" (UID: \"362f1775-4ab7-400a-b216-a22e4ca6c9fb\") " pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:38.380540 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:38.380416 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:22:38.380540 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:38.380473 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs podName:362f1775-4ab7-400a-b216-a22e4ca6c9fb nodeName:}" failed. No retries permitted until 2026-07-06 19:22:39.380460269 +0000 UTC m=+4.009414214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs") pod "network-metrics-daemon-lh7hd" (UID: "362f1775-4ab7-400a-b216-a22e4ca6c9fb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:22:38.480929 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.480894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pp42\" (UniqueName: \"kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42\") pod \"network-check-target-vfp2h\" (UID: \"abb3d52d-bb53-47f0-9708-075b2eae0e2d\") " pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:22:38.481119 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.480970 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret\") pod \"global-pull-secret-syncer-kpsb4\" (UID: \"a6d9fe54-0654-4f9d-bb79-0fbeec14c98f\") " pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:38.481119 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:38.481088 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:22:38.481255 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:38.481112 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:22:38.481255 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:38.481139 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:22:38.481255 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:38.481153 2578 projected.go:196] Error preparing data for projected volume kube-api-access-8pp42 for pod openshift-network-diagnostics/network-check-target-vfp2h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:22:38.481255 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:38.481161 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret podName:a6d9fe54-0654-4f9d-bb79-0fbeec14c98f nodeName:}" failed. No retries permitted until 2026-07-06 19:22:39.481141236 +0000 UTC m=+4.110095196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret") pod "global-pull-secret-syncer-kpsb4" (UID: "a6d9fe54-0654-4f9d-bb79-0fbeec14c98f") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:22:38.481255 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:38.481210 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42 podName:abb3d52d-bb53-47f0-9708-075b2eae0e2d nodeName:}" failed. No retries permitted until 2026-07-06 19:22:39.481191654 +0000 UTC m=+4.110145629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8pp42" (UniqueName: "kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42") pod "network-check-target-vfp2h" (UID: "abb3d52d-bb53-47f0-9708-075b2eae0e2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:22:38.745344 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.745256 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:22:38.864309 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.864224 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:17:38 +0000 UTC" deadline="2028-02-11 11:36:12.964188906 +0000 UTC" Jul 06 19:22:38.864309 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.864300 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14032h13m34.099893306s" Jul 06 19:22:38.881368 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.881327 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c2cs8" event={"ID":"1be1a75c-b7c1-466c-99cb-bdc2e3749d87","Type":"ContainerStarted","Data":"337f10fe505aad09bccc473929759d6bea86f8a3e2241f5ed14ef919d8467b1b"} Jul 06 19:22:38.882620 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.882591 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" event={"ID":"0b2f8393-722b-4d31-b5ad-702a91f1ebc8","Type":"ContainerStarted","Data":"e1096b527b4129772ad49dcd6f598ee4d8e5491d4f941c90b6bbf5493923b44a"} Jul 06 19:22:38.883901 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.883871 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nx2gf" event={"ID":"a56948c7-e923-402c-a264-1ac5dec90393","Type":"ContainerStarted","Data":"ba67606c7030eb60e3d69871d640e31fc16397ddb466c7fa866f5717bf2f67fe"} Jul 06 19:22:38.885172 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.885149 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wcsxd" event={"ID":"63219616-9c15-4abc-90ee-42160928e479","Type":"ContainerStarted","Data":"8b985f8c8ebaa79911ab19e313a90bfcd1e0fc7b945d5e71f7ec3a48b2cfec2e"} Jul 06 19:22:38.886408 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.886359 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wk4zk" event={"ID":"3e4c4a93-9461-4a11-aef0-e1399f9bc611","Type":"ContainerStarted","Data":"4b8393c61ec7fa0b7700c54137886c9cb9a734831c2b660b3d2d9c3aa14af30d"} Jul 06 19:22:38.887627 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.887607 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qfm4f" event={"ID":"ecbc6515-e4c1-489d-ac42-dd3a967b782c","Type":"ContainerStarted","Data":"64450ac91ef85dea93a288f13a233dda89f2829e736984afbd95eb47d05e0bde"} Jul 06 19:22:38.888653 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.888634 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" event={"ID":"140ec7e0-97cd-45df-b179-e62e429073b4","Type":"ContainerStarted","Data":"05eecbebbc6cb1287bfbcf4c1f1959e5af023e94df426dd9314b629773a64a2b"} Jul 06 19:22:38.889811 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:38.889791 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" event={"ID":"363d0018-46e0-4f8c-bf7c-3c582c493c88","Type":"ContainerStarted","Data":"64a369813a404d297591a81684ba578bda6814430b15061883df61d34cb936c0"} Jul 06 19:22:39.388476 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:39.387921 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs\") pod \"network-metrics-daemon-lh7hd\" (UID: \"362f1775-4ab7-400a-b216-a22e4ca6c9fb\") " pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:39.388476 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:39.388364 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:22:39.388476 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:39.388432 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs podName:362f1775-4ab7-400a-b216-a22e4ca6c9fb nodeName:}" failed. No retries permitted until 2026-07-06 19:22:41.388412069 +0000 UTC m=+6.017366018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs") pod "network-metrics-daemon-lh7hd" (UID: "362f1775-4ab7-400a-b216-a22e4ca6c9fb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:22:39.489072 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:39.489031 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret\") pod \"global-pull-secret-syncer-kpsb4\" (UID: \"a6d9fe54-0654-4f9d-bb79-0fbeec14c98f\") " pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:39.489278 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:39.489106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pp42\" (UniqueName: \"kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42\") pod \"network-check-target-vfp2h\" (UID: \"abb3d52d-bb53-47f0-9708-075b2eae0e2d\") " pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:22:39.489355 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:39.489277 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:22:39.489355 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:39.489302 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:22:39.489355 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:39.489319 2578 projected.go:196] Error preparing data for projected volume kube-api-access-8pp42 for pod openshift-network-diagnostics/network-check-target-vfp2h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:22:39.489526 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:39.489387 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42 podName:abb3d52d-bb53-47f0-9708-075b2eae0e2d nodeName:}" failed. No retries permitted until 2026-07-06 19:22:41.489365322 +0000 UTC m=+6.118319276 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8pp42" (UniqueName: "kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42") pod "network-check-target-vfp2h" (UID: "abb3d52d-bb53-47f0-9708-075b2eae0e2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:22:39.489911 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:39.489890 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:22:39.490017 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:39.489951 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret podName:a6d9fe54-0654-4f9d-bb79-0fbeec14c98f nodeName:}" failed. No retries permitted until 2026-07-06 19:22:41.489934246 +0000 UTC m=+6.118888207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret") pod "global-pull-secret-syncer-kpsb4" (UID: "a6d9fe54-0654-4f9d-bb79-0fbeec14c98f") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:22:39.876485 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:39.876402 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:39.876985 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:39.876555 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lh7hd" podUID="362f1775-4ab7-400a-b216-a22e4ca6c9fb" Jul 06 19:22:39.877052 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:39.877005 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:39.877127 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:39.877104 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpsb4" podUID="a6d9fe54-0654-4f9d-bb79-0fbeec14c98f" Jul 06 19:22:39.877190 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:39.877174 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:22:39.877270 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:39.877251 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfp2h" podUID="abb3d52d-bb53-47f0-9708-075b2eae0e2d" Jul 06 19:22:39.877457 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:39.877382 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:17:38 +0000 UTC" deadline="2028-04-11 21:19:02.863064356 +0000 UTC" Jul 06 19:22:39.877457 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:39.877427 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15481h56m22.985643093s" Jul 06 19:22:39.903791 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:39.903744 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal" event={"ID":"aca81e81575313b78c60adfe6a1aa8fd","Type":"ContainerStarted","Data":"577b822b6383ad4fb626e2377da09a4454d294acebfa5812be4658b4155e986e"} Jul 06 19:22:41.406131 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:41.406051 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs\") pod \"network-metrics-daemon-lh7hd\" (UID: \"362f1775-4ab7-400a-b216-a22e4ca6c9fb\") " pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:41.406678 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:41.406225 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:22:41.406678 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:41.406293 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs podName:362f1775-4ab7-400a-b216-a22e4ca6c9fb nodeName:}" failed. No retries permitted until 2026-07-06 19:22:45.406273337 +0000 UTC m=+10.035227294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs") pod "network-metrics-daemon-lh7hd" (UID: "362f1775-4ab7-400a-b216-a22e4ca6c9fb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:22:41.507016 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:41.506977 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret\") pod \"global-pull-secret-syncer-kpsb4\" (UID: \"a6d9fe54-0654-4f9d-bb79-0fbeec14c98f\") " pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:41.507190 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:41.507056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pp42\" (UniqueName: \"kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42\") pod \"network-check-target-vfp2h\" (UID: \"abb3d52d-bb53-47f0-9708-075b2eae0e2d\") " pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:22:41.507264 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:41.507210 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:22:41.507264 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:41.507230 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:22:41.507264 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:41.507245 2578 projected.go:196] Error preparing data for projected volume kube-api-access-8pp42 for pod openshift-network-diagnostics/network-check-target-vfp2h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:22:41.507429 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:41.507305 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42 podName:abb3d52d-bb53-47f0-9708-075b2eae0e2d nodeName:}" failed. No retries permitted until 2026-07-06 19:22:45.507283712 +0000 UTC m=+10.136237679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8pp42" (UniqueName: "kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42") pod "network-check-target-vfp2h" (UID: "abb3d52d-bb53-47f0-9708-075b2eae0e2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:22:41.507727 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:41.507710 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:22:41.507792 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:41.507768 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret podName:a6d9fe54-0654-4f9d-bb79-0fbeec14c98f nodeName:}" failed. No retries permitted until 2026-07-06 19:22:45.507753016 +0000 UTC m=+10.136706965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret") pod "global-pull-secret-syncer-kpsb4" (UID: "a6d9fe54-0654-4f9d-bb79-0fbeec14c98f") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:22:41.869918 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:41.869828 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:41.870073 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:41.869836 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:41.870073 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:41.869967 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lh7hd" podUID="362f1775-4ab7-400a-b216-a22e4ca6c9fb" Jul 06 19:22:41.870073 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:41.869979 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:22:41.870227 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:41.870069 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpsb4" podUID="a6d9fe54-0654-4f9d-bb79-0fbeec14c98f" Jul 06 19:22:41.870227 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:41.870147 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfp2h" podUID="abb3d52d-bb53-47f0-9708-075b2eae0e2d" Jul 06 19:22:43.871455 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:43.870547 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:43.871455 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:43.870684 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpsb4" podUID="a6d9fe54-0654-4f9d-bb79-0fbeec14c98f" Jul 06 19:22:43.871455 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:43.871132 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:22:43.871455 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:43.871227 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfp2h" podUID="abb3d52d-bb53-47f0-9708-075b2eae0e2d" Jul 06 19:22:43.871455 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:43.871299 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:43.871455 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:43.871383 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lh7hd" podUID="362f1775-4ab7-400a-b216-a22e4ca6c9fb" Jul 06 19:22:45.443467 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:45.443430 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs\") pod \"network-metrics-daemon-lh7hd\" (UID: \"362f1775-4ab7-400a-b216-a22e4ca6c9fb\") " pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:45.443991 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:45.443635 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:22:45.443991 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:45.443713 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs podName:362f1775-4ab7-400a-b216-a22e4ca6c9fb nodeName:}" failed. No retries permitted until 2026-07-06 19:22:53.443681041 +0000 UTC m=+18.072635002 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs") pod "network-metrics-daemon-lh7hd" (UID: "362f1775-4ab7-400a-b216-a22e4ca6c9fb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:22:45.544591 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:45.544298 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret\") pod \"global-pull-secret-syncer-kpsb4\" (UID: \"a6d9fe54-0654-4f9d-bb79-0fbeec14c98f\") " pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:45.544591 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:45.544374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pp42\" (UniqueName: \"kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42\") pod \"network-check-target-vfp2h\" (UID: \"abb3d52d-bb53-47f0-9708-075b2eae0e2d\") " pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:22:45.544591 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:45.544501 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:22:45.544591 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:45.544513 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:22:45.544591 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:45.544531 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:22:45.544591 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:45.544544 2578 projected.go:196] Error preparing data for projected volume kube-api-access-8pp42 for pod openshift-network-diagnostics/network-check-target-vfp2h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:22:45.544591 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:45.544567 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret podName:a6d9fe54-0654-4f9d-bb79-0fbeec14c98f nodeName:}" failed. No retries permitted until 2026-07-06 19:22:53.544549193 +0000 UTC m=+18.173503159 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret") pod "global-pull-secret-syncer-kpsb4" (UID: "a6d9fe54-0654-4f9d-bb79-0fbeec14c98f") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:22:45.544591 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:45.544585 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42 podName:abb3d52d-bb53-47f0-9708-075b2eae0e2d nodeName:}" failed. No retries permitted until 2026-07-06 19:22:53.544576167 +0000 UTC m=+18.173530117 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8pp42" (UniqueName: "kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42") pod "network-check-target-vfp2h" (UID: "abb3d52d-bb53-47f0-9708-075b2eae0e2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:22:45.871057 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:45.870785 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:45.871057 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:45.870909 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:45.871057 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:45.870923 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lh7hd" podUID="362f1775-4ab7-400a-b216-a22e4ca6c9fb" Jul 06 19:22:45.871057 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:45.871041 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpsb4" podUID="a6d9fe54-0654-4f9d-bb79-0fbeec14c98f" Jul 06 19:22:45.871394 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:45.871108 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:22:45.871394 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:45.871216 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfp2h" podUID="abb3d52d-bb53-47f0-9708-075b2eae0e2d" Jul 06 19:22:47.870220 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:47.869708 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:47.870220 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:47.869758 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:22:47.870220 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:47.869709 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:47.870220 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:47.869860 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lh7hd" podUID="362f1775-4ab7-400a-b216-a22e4ca6c9fb" Jul 06 19:22:47.870220 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:47.869954 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpsb4" podUID="a6d9fe54-0654-4f9d-bb79-0fbeec14c98f" Jul 06 19:22:47.870220 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:47.870024 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfp2h" podUID="abb3d52d-bb53-47f0-9708-075b2eae0e2d" Jul 06 19:22:49.869647 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:49.869606 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:49.869647 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:49.869634 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:49.870178 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:49.869606 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:22:49.870178 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:49.869784 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lh7hd" podUID="362f1775-4ab7-400a-b216-a22e4ca6c9fb" Jul 06 19:22:49.870178 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:49.869884 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpsb4" podUID="a6d9fe54-0654-4f9d-bb79-0fbeec14c98f" Jul 06 19:22:49.870178 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:49.870016 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfp2h" podUID="abb3d52d-bb53-47f0-9708-075b2eae0e2d" Jul 06 19:22:51.868962 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:51.868925 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:22:51.869412 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:51.868925 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:51.869668 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:51.869555 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:51.869668 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:51.869564 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfp2h" podUID="abb3d52d-bb53-47f0-9708-075b2eae0e2d" Jul 06 19:22:51.870259 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:51.870219 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lh7hd" podUID="362f1775-4ab7-400a-b216-a22e4ca6c9fb" Jul 06 19:22:51.870407 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:51.870384 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpsb4" podUID="a6d9fe54-0654-4f9d-bb79-0fbeec14c98f" Jul 06 19:22:53.506789 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:53.506705 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs\") pod \"network-metrics-daemon-lh7hd\" (UID: \"362f1775-4ab7-400a-b216-a22e4ca6c9fb\") " pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:53.507257 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:53.506812 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:22:53.507257 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:53.506883 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs podName:362f1775-4ab7-400a-b216-a22e4ca6c9fb nodeName:}" failed. No retries permitted until 2026-07-06 19:23:09.506862324 +0000 UTC m=+34.135816270 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs") pod "network-metrics-daemon-lh7hd" (UID: "362f1775-4ab7-400a-b216-a22e4ca6c9fb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:22:53.608084 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:53.608048 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret\") pod \"global-pull-secret-syncer-kpsb4\" (UID: \"a6d9fe54-0654-4f9d-bb79-0fbeec14c98f\") " pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:53.608264 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:53.608112 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pp42\" (UniqueName: \"kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42\") pod \"network-check-target-vfp2h\" (UID: \"abb3d52d-bb53-47f0-9708-075b2eae0e2d\") " pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:22:53.608264 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:53.608200 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:22:53.608264 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:53.608256 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:22:53.608420 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:53.608269 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret podName:a6d9fe54-0654-4f9d-bb79-0fbeec14c98f nodeName:}" failed. No retries permitted until 2026-07-06 19:23:09.608251405 +0000 UTC m=+34.237205364 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret") pod "global-pull-secret-syncer-kpsb4" (UID: "a6d9fe54-0654-4f9d-bb79-0fbeec14c98f") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:22:53.608420 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:53.608275 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:22:53.608420 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:53.608288 2578 projected.go:196] Error preparing data for projected volume kube-api-access-8pp42 for pod openshift-network-diagnostics/network-check-target-vfp2h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:22:53.608420 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:53.608333 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42 podName:abb3d52d-bb53-47f0-9708-075b2eae0e2d nodeName:}" failed. No retries permitted until 2026-07-06 19:23:09.60831733 +0000 UTC m=+34.237271275 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8pp42" (UniqueName: "kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42") pod "network-check-target-vfp2h" (UID: "abb3d52d-bb53-47f0-9708-075b2eae0e2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:22:53.869641 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:53.869606 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:22:53.869641 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:53.869644 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:53.869921 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:53.869611 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:53.869921 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:53.869765 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfp2h" podUID="abb3d52d-bb53-47f0-9708-075b2eae0e2d" Jul 06 19:22:53.869921 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:53.869827 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lh7hd" podUID="362f1775-4ab7-400a-b216-a22e4ca6c9fb" Jul 06 19:22:53.870028 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:53.869924 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpsb4" podUID="a6d9fe54-0654-4f9d-bb79-0fbeec14c98f" Jul 06 19:22:55.869763 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:55.869727 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:55.870206 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:55.869843 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:22:55.870206 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:55.869847 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lh7hd" podUID="362f1775-4ab7-400a-b216-a22e4ca6c9fb" Jul 06 19:22:55.870206 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:55.869909 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfp2h" podUID="abb3d52d-bb53-47f0-9708-075b2eae0e2d" Jul 06 19:22:55.870206 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:55.869941 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:55.870206 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:55.870007 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpsb4" podUID="a6d9fe54-0654-4f9d-bb79-0fbeec14c98f" Jul 06 19:22:57.869503 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:57.869333 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:57.870467 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:57.869334 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:57.870534 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:57.870493 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lh7hd" podUID="362f1775-4ab7-400a-b216-a22e4ca6c9fb" Jul 06 19:22:57.870534 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:57.869928 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpsb4" podUID="a6d9fe54-0654-4f9d-bb79-0fbeec14c98f" Jul 06 19:22:57.870534 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:57.869396 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:22:57.870717 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:57.870626 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfp2h" podUID="abb3d52d-bb53-47f0-9708-075b2eae0e2d" Jul 06 19:22:57.956112 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:57.956081 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" event={"ID":"0b2f8393-722b-4d31-b5ad-702a91f1ebc8","Type":"ContainerStarted","Data":"7d0546ed28a737d99c78d9c364b770f2f4ae247dcd1e5d064e004943292403d3"} Jul 06 19:22:57.956274 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:57.956165 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" event={"ID":"0b2f8393-722b-4d31-b5ad-702a91f1ebc8","Type":"ContainerStarted","Data":"52ab5de88a400d20e25f37369b56da15fdc5e0c9bf58e05f124d9388b6974d0a"} Jul 06 19:22:57.956274 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:57.956210 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" event={"ID":"0b2f8393-722b-4d31-b5ad-702a91f1ebc8","Type":"ContainerStarted","Data":"ac0f747d25f49d50c6a8ad715658d9194d02f8aeefcb8bd88d14aa6d958a4a99"} Jul 06 19:22:57.957403 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:57.957374 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wcsxd" event={"ID":"63219616-9c15-4abc-90ee-42160928e479","Type":"ContainerStarted","Data":"d4897424a7caeeaacec1ea82eadf84ea981ccc5d1294e16ddc8fa7355a11cb63"} Jul 06 19:22:57.958612 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:57.958590 2578 generic.go:356] "Generic (PLEG): container finished" podID="cf1609911ca9f346fada711322198d5a" containerID="e720629af992d18cb9cc175da798e21c94840dfc1ad193af54e3d211f19be439" exitCode=0 Jul 06 19:22:57.958732 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:57.958675 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal" event={"ID":"cf1609911ca9f346fada711322198d5a","Type":"ContainerDied","Data":"e720629af992d18cb9cc175da798e21c94840dfc1ad193af54e3d211f19be439"} Jul 06 19:22:57.960016 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:57.959997 2578 generic.go:356] "Generic (PLEG): container finished" podID="3e4c4a93-9461-4a11-aef0-e1399f9bc611" containerID="dd3da5ed307afe46e7cde2d720cc3525dbaacd09570839ad65587f327af7dbb8" exitCode=0 Jul 06 19:22:57.960092 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:57.960066 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wk4zk" event={"ID":"3e4c4a93-9461-4a11-aef0-e1399f9bc611","Type":"ContainerDied","Data":"dd3da5ed307afe46e7cde2d720cc3525dbaacd09570839ad65587f327af7dbb8"} Jul 06 19:22:57.961509 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:57.961492 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qfm4f" event={"ID":"ecbc6515-e4c1-489d-ac42-dd3a967b782c","Type":"ContainerStarted","Data":"115630f8d4fb5cccef013ff7a9a5eb9dc8f25cdf3180a26a00dc8e1934dd4eaf"} Jul 06 19:22:57.962851 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:57.962832 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" event={"ID":"140ec7e0-97cd-45df-b179-e62e429073b4","Type":"ContainerStarted","Data":"a5de256f12083457deb91a90819962e70a1c2d1672e69e323efd711fdb7a901f"} Jul 06 19:22:57.964212 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:57.964195 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" event={"ID":"363d0018-46e0-4f8c-bf7c-3c582c493c88","Type":"ContainerStarted","Data":"e0bfbbc8b6bf218a72153073bb8a22cc6cb4bfd41f0d6f1e1b1e1ab87e36fd41"} Jul 06 19:22:57.965581 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:57.965563 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c2cs8" event={"ID":"1be1a75c-b7c1-466c-99cb-bdc2e3749d87","Type":"ContainerStarted","Data":"a313a525424ac11f4bc21b485c84b102ca13f6a8f2e318c961ca93e0bd2f590c"} Jul 06 19:22:57.997155 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:57.997058 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal" podStartSLOduration=20.997023466 podStartE2EDuration="20.997023466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:39.937808836 +0000 UTC m=+4.566762809" watchObservedRunningTime="2026-07-06 19:22:57.997023466 +0000 UTC m=+22.625977414" Jul 06 19:22:57.997559 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:57.997528 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wcsxd" podStartSLOduration=6.938080648 podStartE2EDuration="21.997515753s" totalImagesPullingTime="15.059435105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:57.997284723 +0000 UTC m=+22.626238692" watchObservedRunningTime="2026-07-06 19:22:57.997515753 +0000 UTC m=+22.626469719" Jul 06 19:22:58.103281 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:58.103083 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pg5z7" podStartSLOduration=2.699348389 podStartE2EDuration="22.103067539s" totalImagesPullingTime="19.40371915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:58.101975808 +0000 UTC m=+22.730929773" watchObservedRunningTime="2026-07-06 19:22:58.103067539 +0000 UTC m=+22.732021505" Jul 06 19:22:58.133314 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:58.133249 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qfm4f" podStartSLOduration=7.103029519 podStartE2EDuration="22.133231983s" totalImagesPullingTime="15.030202464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:58.132412912 +0000 UTC m=+22.761366879" watchObservedRunningTime="2026-07-06 19:22:58.133231983 +0000 UTC m=+22.762185953" Jul 06 19:22:58.317896 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:58.317846 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c2cs8" podStartSLOduration=2.917001785 podStartE2EDuration="22.317828163s" totalImagesPullingTime="19.400826378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:58.316937905 +0000 UTC m=+22.945891870" watchObservedRunningTime="2026-07-06 19:22:58.317828163 +0000 UTC m=+22.946782130" Jul 06 19:22:58.972365 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:58.972319 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" event={"ID":"0b2f8393-722b-4d31-b5ad-702a91f1ebc8","Type":"ContainerStarted","Data":"71c749a290127725e3cdc691a4fb49da50cbcabeb78c8322880a7b6a15c06663"} Jul 06 19:22:58.972896 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:58.972798 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" event={"ID":"0b2f8393-722b-4d31-b5ad-702a91f1ebc8","Type":"ContainerStarted","Data":"ced2a9bdea9647038419b01014c6f7a6aa0c222031b5267a5961d4ca6d7227bf"} Jul 06 19:22:58.972986 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:58.972953 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" event={"ID":"0b2f8393-722b-4d31-b5ad-702a91f1ebc8","Type":"ContainerStarted","Data":"7e2c95e92a8de3fdf72a40992303f0cf0ee77d92d98369ca675bd1be03007b9c"} Jul 06 19:22:58.976227 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:58.976193 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal" event={"ID":"cf1609911ca9f346fada711322198d5a","Type":"ContainerStarted","Data":"9ad2e44ad6cc16578a0f135c402d7d484eb220c833dd46e199fa6e8df9db5341"} Jul 06 19:22:59.028402 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:59.028352 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal" podStartSLOduration=22.028337358 podStartE2EDuration="22.028337358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:59.028129372 +0000 UTC m=+23.657083339" watchObservedRunningTime="2026-07-06 19:22:59.028337358 +0000 UTC m=+23.657291323" Jul 06 19:22:59.579261 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:59.579232 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 19:22:59.827285 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:59.827184 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T19:22:59.579255133Z","UUID":"57b4bd92-3ab4-405b-8cdc-7fef1c4ce405","Handler":null,"Name":"","Endpoint":""} Jul 06 19:22:59.828763 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:59.828740 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 19:22:59.828886 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:59.828785 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 19:22:59.869139 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:59.869109 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:22:59.869295 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:59.869114 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:22:59.869295 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:59.869233 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lh7hd" podUID="362f1775-4ab7-400a-b216-a22e4ca6c9fb" Jul 06 19:22:59.869295 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:59.869256 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:22:59.869436 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:59.869316 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpsb4" podUID="a6d9fe54-0654-4f9d-bb79-0fbeec14c98f" Jul 06 19:22:59.869483 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:22:59.869432 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfp2h" podUID="abb3d52d-bb53-47f0-9708-075b2eae0e2d" Jul 06 19:22:59.980029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:59.979992 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nx2gf" event={"ID":"a56948c7-e923-402c-a264-1ac5dec90393","Type":"ContainerStarted","Data":"e9d37657e8842accde2b864a41ba972f6db2906777006b25d03b90173a8f7ce8"} Jul 06 19:22:59.981841 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:22:59.981816 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" event={"ID":"140ec7e0-97cd-45df-b179-e62e429073b4","Type":"ContainerStarted","Data":"2c9c70e8d37085a48c95b519af7323138bb798c57cd5712fa004d6f2e5ff07c7"} Jul 06 19:23:00.015426 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:00.015363 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nx2gf" podStartSLOduration=4.635659548 podStartE2EDuration="24.015345639s" totalImagesPullingTime="19.379686091s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:00.00908961 +0000 UTC m=+24.638043588" watchObservedRunningTime="2026-07-06 19:23:00.015345639 +0000 UTC m=+24.644299606" Jul 06 19:23:00.988397 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:00.988368 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" event={"ID":"0b2f8393-722b-4d31-b5ad-702a91f1ebc8","Type":"ContainerStarted","Data":"8f9d042a9aab5ce0f65221056b833ba66dd002a1b39e57a72ce1634066416399"} Jul 06 19:23:01.214917 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:01.214831 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wcsxd" Jul 06 19:23:01.215513 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:01.215488 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wcsxd" Jul 06 19:23:01.869444 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:01.869404 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:23:01.869896 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:01.869418 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:23:01.869896 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:01.869532 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfp2h" podUID="abb3d52d-bb53-47f0-9708-075b2eae0e2d" Jul 06 19:23:01.869896 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:01.869620 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpsb4" podUID="a6d9fe54-0654-4f9d-bb79-0fbeec14c98f" Jul 06 19:23:01.869896 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:01.869418 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:23:01.869896 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:01.869731 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lh7hd" podUID="362f1775-4ab7-400a-b216-a22e4ca6c9fb" Jul 06 19:23:01.992003 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:01.991968 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" event={"ID":"140ec7e0-97cd-45df-b179-e62e429073b4","Type":"ContainerStarted","Data":"d6904cdedd5ce50ac646476bb22926cf958717d5f6495a3a57071fea64863b5f"} Jul 06 19:23:01.992475 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:01.992122 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wcsxd" Jul 06 19:23:01.992649 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:01.992634 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wcsxd" Jul 06 19:23:02.039252 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:02.039204 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9qhn" podStartSLOduration=3.748325559 podStartE2EDuration="26.039187483s" totalImagesPullingTime="22.290861924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:02.033511225 +0000 UTC m=+26.662465203" watchObservedRunningTime="2026-07-06 19:23:02.039187483 +0000 UTC m=+26.668141450" Jul 06 19:23:02.999535 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:02.999282 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" event={"ID":"0b2f8393-722b-4d31-b5ad-702a91f1ebc8","Type":"ContainerStarted","Data":"cd2a6a396cd2212773eaf6112624a634bb789173de15e955e57c8ab375871ea3"} Jul 06 19:23:03.030540 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:03.030493 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" podStartSLOduration=7.592880875 podStartE2EDuration="27.030475426s" totalImagesPullingTime="19.437594551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:03.029774248 +0000 UTC m=+27.658728216" watchObservedRunningTime="2026-07-06 19:23:03.030475426 +0000 UTC m=+27.659429392" Jul 06 19:23:03.869348 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:03.869309 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:23:03.869517 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:03.869308 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:23:03.869517 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:03.869458 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lh7hd" podUID="362f1775-4ab7-400a-b216-a22e4ca6c9fb" Jul 06 19:23:03.869636 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:03.869322 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:23:03.869636 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:03.869516 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfp2h" podUID="abb3d52d-bb53-47f0-9708-075b2eae0e2d" Jul 06 19:23:03.869636 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:03.869611 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpsb4" podUID="a6d9fe54-0654-4f9d-bb79-0fbeec14c98f" Jul 06 19:23:04.002334 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:04.002299 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:23:04.002820 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:04.002801 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:23:04.002908 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:04.002859 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:23:04.021335 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:04.021306 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:23:04.021511 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:04.021498 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:23:05.029496 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:05.029459 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kpsb4"] Jul 06 19:23:05.030145 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:05.029589 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:23:05.030145 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:05.029722 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpsb4" podUID="a6d9fe54-0654-4f9d-bb79-0fbeec14c98f" Jul 06 19:23:05.033976 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:05.033949 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vfp2h"] Jul 06 19:23:05.034124 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:05.034064 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:23:05.034186 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:05.034166 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfp2h" podUID="abb3d52d-bb53-47f0-9708-075b2eae0e2d" Jul 06 19:23:05.034615 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:05.034592 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lh7hd"] Jul 06 19:23:05.034738 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:05.034711 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:23:05.034838 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:05.034817 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lh7hd" podUID="362f1775-4ab7-400a-b216-a22e4ca6c9fb" Jul 06 19:23:06.008283 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:06.008109 2578 generic.go:356] "Generic (PLEG): container finished" podID="3e4c4a93-9461-4a11-aef0-e1399f9bc611" containerID="5aa8423d18f7d9df7dff36bf1c74250a916644877aca524e907c3fdcd49394ed" exitCode=0 Jul 06 19:23:06.008457 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:06.008196 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wk4zk" event={"ID":"3e4c4a93-9461-4a11-aef0-e1399f9bc611","Type":"ContainerDied","Data":"5aa8423d18f7d9df7dff36bf1c74250a916644877aca524e907c3fdcd49394ed"} Jul 06 19:23:06.868984 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:06.868954 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:23:06.869356 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:06.868955 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:23:06.869356 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:06.868962 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:23:06.869356 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:06.869112 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpsb4" podUID="a6d9fe54-0654-4f9d-bb79-0fbeec14c98f" Jul 06 19:23:06.869356 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:06.869242 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfp2h" podUID="abb3d52d-bb53-47f0-9708-075b2eae0e2d" Jul 06 19:23:06.869356 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:06.869331 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lh7hd" podUID="362f1775-4ab7-400a-b216-a22e4ca6c9fb" Jul 06 19:23:07.012871 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:07.012840 2578 generic.go:356] "Generic (PLEG): container finished" podID="3e4c4a93-9461-4a11-aef0-e1399f9bc611" containerID="7dff3af55239c3f9a5ae9d1c657d964c12e08c6c5fe0ebd9d092722ed179d70c" exitCode=0 Jul 06 19:23:07.013010 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:07.012888 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wk4zk" event={"ID":"3e4c4a93-9461-4a11-aef0-e1399f9bc611","Type":"ContainerDied","Data":"7dff3af55239c3f9a5ae9d1c657d964c12e08c6c5fe0ebd9d092722ed179d70c"} Jul 06 19:23:08.017329 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:08.017303 2578 generic.go:356] "Generic (PLEG): container finished" podID="3e4c4a93-9461-4a11-aef0-e1399f9bc611" containerID="04c312e3abe84c7cbaeb7dc1a75e1d0343485d03f0ef3ba4c524ebbd65fe26fb" exitCode=0 Jul 06 19:23:08.017689 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:08.017349 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wk4zk" event={"ID":"3e4c4a93-9461-4a11-aef0-e1399f9bc611","Type":"ContainerDied","Data":"04c312e3abe84c7cbaeb7dc1a75e1d0343485d03f0ef3ba4c524ebbd65fe26fb"} Jul 06 19:23:08.869714 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:08.869667 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:23:08.869714 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:08.869716 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:23:08.869947 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:08.869776 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:23:08.869947 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:08.869791 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lh7hd" podUID="362f1775-4ab7-400a-b216-a22e4ca6c9fb" Jul 06 19:23:08.869947 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:08.869857 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfp2h" podUID="abb3d52d-bb53-47f0-9708-075b2eae0e2d" Jul 06 19:23:08.869947 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:08.869897 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpsb4" podUID="a6d9fe54-0654-4f9d-bb79-0fbeec14c98f" Jul 06 19:23:09.520056 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:09.520016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs\") pod \"network-metrics-daemon-lh7hd\" (UID: \"362f1775-4ab7-400a-b216-a22e4ca6c9fb\") " pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:23:09.520461 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:09.520169 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:09.520461 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:09.520239 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs podName:362f1775-4ab7-400a-b216-a22e4ca6c9fb nodeName:}" failed. No retries permitted until 2026-07-06 19:23:41.520224014 +0000 UTC m=+66.149177962 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs") pod "network-metrics-daemon-lh7hd" (UID: "362f1775-4ab7-400a-b216-a22e4ca6c9fb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:09.620797 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:09.620764 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret\") pod \"global-pull-secret-syncer-kpsb4\" (UID: \"a6d9fe54-0654-4f9d-bb79-0fbeec14c98f\") " pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:23:09.620935 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:09.620815 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pp42\" (UniqueName: \"kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42\") pod \"network-check-target-vfp2h\" (UID: \"abb3d52d-bb53-47f0-9708-075b2eae0e2d\") " pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:23:09.620935 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:09.620924 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:09.621022 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:09.620939 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:23:09.621022 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:09.620953 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:23:09.621022 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:09.620965 2578 projected.go:196] Error preparing data for projected volume kube-api-access-8pp42 for pod openshift-network-diagnostics/network-check-target-vfp2h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:09.621022 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:09.620992 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret podName:a6d9fe54-0654-4f9d-bb79-0fbeec14c98f nodeName:}" failed. No retries permitted until 2026-07-06 19:23:41.620974394 +0000 UTC m=+66.249928359 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret") pod "global-pull-secret-syncer-kpsb4" (UID: "a6d9fe54-0654-4f9d-bb79-0fbeec14c98f") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:09.621022 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:09.621008 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42 podName:abb3d52d-bb53-47f0-9708-075b2eae0e2d nodeName:}" failed. No retries permitted until 2026-07-06 19:23:41.621001605 +0000 UTC m=+66.249955560 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8pp42" (UniqueName: "kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42") pod "network-check-target-vfp2h" (UID: "abb3d52d-bb53-47f0-9708-075b2eae0e2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:10.869574 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:10.869544 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:23:10.870039 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:10.869549 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:23:10.870039 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:10.869547 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:23:10.870039 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:10.869667 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vfp2h" podUID="abb3d52d-bb53-47f0-9708-075b2eae0e2d" Jul 06 19:23:10.870039 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:10.869776 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lh7hd" podUID="362f1775-4ab7-400a-b216-a22e4ca6c9fb" Jul 06 19:23:10.870039 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:10.869860 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kpsb4" podUID="a6d9fe54-0654-4f9d-bb79-0fbeec14c98f" Jul 06 19:23:11.307262 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.307236 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeReady" Jul 06 19:23:11.307428 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.307384 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 19:23:11.349660 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.349608 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-954dc4658-xvp8b"] Jul 06 19:23:11.355465 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.355089 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.356378 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.356357 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-xgcrt"] Jul 06 19:23:11.358322 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.358218 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jf8st\"" Jul 06 19:23:11.358824 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.358639 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 19:23:11.358824 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.358711 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 19:23:11.360300 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.360281 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 19:23:11.362227 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.362092 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc"] Jul 06 19:23:11.362477 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.362459 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.367310 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.367279 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4jb65\"" Jul 06 19:23:11.367568 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.367550 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 06 19:23:11.367928 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.367906 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 19:23:11.368058 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.368042 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 06 19:23:11.368292 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.368272 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 19:23:11.369995 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.369980 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 19:23:11.373797 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.373778 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-68d9d48ccc-kk7pz"] Jul 06 19:23:11.373925 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.373905 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" Jul 06 19:23:11.375140 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.375122 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 06 19:23:11.377308 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.377280 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 06 19:23:11.377308 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.377324 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 19:23:11.377527 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.377448 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mv8tf\"" Jul 06 19:23:11.379805 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.379785 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc"] Jul 06 19:23:11.379928 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.379904 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:11.382397 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.382316 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 06 19:23:11.382918 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.382900 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 19:23:11.384015 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.383994 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-954dc4658-xvp8b"] Jul 06 19:23:11.384607 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.384584 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dk4kn"] Jul 06 19:23:11.385238 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.385086 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 06 19:23:11.385414 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.385390 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 06 19:23:11.385657 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.385637 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 06 19:23:11.385787 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.385752 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rbhnq\"" Jul 06 19:23:11.386086 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.386066 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 06 19:23:11.387620 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.387599 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 19:23:11.387736 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.387627 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 19:23:11.391770 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.391754 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68d9d48ccc-kk7pz"] Jul 06 19:23:11.391865 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.391834 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dk4kn" Jul 06 19:23:11.394871 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.394851 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 19:23:11.395228 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.395214 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 19:23:11.400457 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.400442 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 19:23:11.400558 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.400545 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qx2jx\"" Jul 06 19:23:11.409678 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.409611 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dk4kn"] Jul 06 19:23:11.410554 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.410526 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-xgcrt"] Jul 06 19:23:11.436287 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.436256 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgwmv\" (UniqueName: \"kubernetes.io/projected/7bf04227-6bc5-4797-a189-f1c12841bfc4-kube-api-access-fgwmv\") pod \"insights-operator-5c847dcfc4-xgcrt\" (UID: \"7bf04227-6bc5-4797-a189-f1c12841bfc4\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.436439 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.436297 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee43e118-575d-4aa1-9276-f9f1ed3606a4-image-registry-private-configuration\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.436439 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.436355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7bf04227-6bc5-4797-a189-f1c12841bfc4-snapshots\") pod \"insights-operator-5c847dcfc4-xgcrt\" (UID: \"7bf04227-6bc5-4797-a189-f1c12841bfc4\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.436439 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.436405 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-bound-sa-token\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.436439 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.436426 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee43e118-575d-4aa1-9276-f9f1ed3606a4-ca-trust-extracted\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.436625 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.436447 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bf04227-6bc5-4797-a189-f1c12841bfc4-tmp\") pod \"insights-operator-5c847dcfc4-xgcrt\" (UID: \"7bf04227-6bc5-4797-a189-f1c12841bfc4\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.436625 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.436463 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7bf04227-6bc5-4797-a189-f1c12841bfc4-serving-cert\") pod \"insights-operator-5c847dcfc4-xgcrt\" (UID: \"7bf04227-6bc5-4797-a189-f1c12841bfc4\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.436625 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.436527 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee43e118-575d-4aa1-9276-f9f1ed3606a4-trusted-ca\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.436625 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.436562 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bf04227-6bc5-4797-a189-f1c12841bfc4-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-xgcrt\" (UID: \"7bf04227-6bc5-4797-a189-f1c12841bfc4\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.436625 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.436588 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.436877 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.436653 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee43e118-575d-4aa1-9276-f9f1ed3606a4-installation-pull-secrets\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.436877 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.436686 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bf04227-6bc5-4797-a189-f1c12841bfc4-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-xgcrt\" (UID: \"7bf04227-6bc5-4797-a189-f1c12841bfc4\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.436877 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.436730 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-certificates\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.436877 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.436756 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrmb8\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-kube-api-access-vrmb8\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.494881 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.494844 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8j849"] Jul 06 19:23:11.501853 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.501826 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8j849" Jul 06 19:23:11.504099 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.504069 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 19:23:11.505601 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.505580 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 19:23:11.505763 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.505615 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bhtlv\"" Jul 06 19:23:11.505878 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.505658 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 19:23:11.505878 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.505659 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 19:23:11.513540 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.513519 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8j849"] Jul 06 19:23:11.537275 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537243 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-bound-sa-token\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.537428 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537292 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49nfp\" (UniqueName: \"kubernetes.io/projected/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-kube-api-access-49nfp\") pod \"cluster-monitoring-operator-64786d9cbb-z8zqc\" (UID: \"8aa685e5-fc07-4d7b-bb90-506c9c9f34be\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" Jul 06 19:23:11.537428 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee43e118-575d-4aa1-9276-f9f1ed3606a4-ca-trust-extracted\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.537544 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537439 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07c083da-b946-4438-afa0-3c5be0f77b9b-cert\") pod \"ingress-canary-dk4kn\" (UID: \"07c083da-b946-4438-afa0-3c5be0f77b9b\") " pod="openshift-ingress-canary/ingress-canary-dk4kn" Jul 06 19:23:11.537544 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537467 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bf04227-6bc5-4797-a189-f1c12841bfc4-tmp\") pod \"insights-operator-5c847dcfc4-xgcrt\" (UID: \"7bf04227-6bc5-4797-a189-f1c12841bfc4\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.537544 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7bf04227-6bc5-4797-a189-f1c12841bfc4-serving-cert\") pod \"insights-operator-5c847dcfc4-xgcrt\" (UID: \"7bf04227-6bc5-4797-a189-f1c12841bfc4\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.537544 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-stats-auth\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:11.537738 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537545 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee43e118-575d-4aa1-9276-f9f1ed3606a4-trusted-ca\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.537738 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537569 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66rrx\" (UniqueName: \"kubernetes.io/projected/07c083da-b946-4438-afa0-3c5be0f77b9b-kube-api-access-66rrx\") pod \"ingress-canary-dk4kn\" (UID: \"07c083da-b946-4438-afa0-3c5be0f77b9b\") " pod="openshift-ingress-canary/ingress-canary-dk4kn" Jul 06 19:23:11.537738 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537597 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bf04227-6bc5-4797-a189-f1c12841bfc4-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-xgcrt\" (UID: \"7bf04227-6bc5-4797-a189-f1c12841bfc4\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.537738 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537626 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.537738 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537681 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee43e118-575d-4aa1-9276-f9f1ed3606a4-installation-pull-secrets\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.537738 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537710 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-default-certificate\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:11.537738 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537737 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:11.538091 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537753 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bf04227-6bc5-4797-a189-f1c12841bfc4-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-xgcrt\" (UID: \"7bf04227-6bc5-4797-a189-f1c12841bfc4\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.538091 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537771 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-certificates\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.538091 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrmb8\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-kube-api-access-vrmb8\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.538091 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgwmv\" (UniqueName: \"kubernetes.io/projected/7bf04227-6bc5-4797-a189-f1c12841bfc4-kube-api-access-fgwmv\") pod \"insights-operator-5c847dcfc4-xgcrt\" (UID: \"7bf04227-6bc5-4797-a189-f1c12841bfc4\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.538091 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537830 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee43e118-575d-4aa1-9276-f9f1ed3606a4-image-registry-private-configuration\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.538091 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-z8zqc\" (UID: \"8aa685e5-fc07-4d7b-bb90-506c9c9f34be\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" Jul 06 19:23:11.538091 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537861 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svwf8\" (UniqueName: \"kubernetes.io/projected/cc56f897-5d59-4451-a517-5bfea063ce0c-kube-api-access-svwf8\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:11.538091 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7bf04227-6bc5-4797-a189-f1c12841bfc4-snapshots\") pod \"insights-operator-5c847dcfc4-xgcrt\" (UID: \"7bf04227-6bc5-4797-a189-f1c12841bfc4\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.538091 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537899 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:11.538091 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.537925 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-z8zqc\" (UID: \"8aa685e5-fc07-4d7b-bb90-506c9c9f34be\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" Jul 06 19:23:11.538537 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.538120 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee43e118-575d-4aa1-9276-f9f1ed3606a4-ca-trust-extracted\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.538537 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.538297 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bf04227-6bc5-4797-a189-f1c12841bfc4-tmp\") pod \"insights-operator-5c847dcfc4-xgcrt\" (UID: \"7bf04227-6bc5-4797-a189-f1c12841bfc4\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.538624 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.538584 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-certificates\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.538890 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:11.538723 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:23:11.538890 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:11.538744 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-954dc4658-xvp8b: secret "image-registry-tls" not found Jul 06 19:23:11.538890 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:11.538810 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls podName:ee43e118-575d-4aa1-9276-f9f1ed3606a4 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:12.038791262 +0000 UTC m=+36.667745210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls") pod "image-registry-954dc4658-xvp8b" (UID: "ee43e118-575d-4aa1-9276-f9f1ed3606a4") : secret "image-registry-tls" not found Jul 06 19:23:11.539110 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.539065 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee43e118-575d-4aa1-9276-f9f1ed3606a4-trusted-ca\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.539110 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.539083 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bf04227-6bc5-4797-a189-f1c12841bfc4-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-xgcrt\" (UID: \"7bf04227-6bc5-4797-a189-f1c12841bfc4\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.539948 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.539462 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7bf04227-6bc5-4797-a189-f1c12841bfc4-snapshots\") pod \"insights-operator-5c847dcfc4-xgcrt\" (UID: \"7bf04227-6bc5-4797-a189-f1c12841bfc4\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.539948 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.539907 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bf04227-6bc5-4797-a189-f1c12841bfc4-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-xgcrt\" (UID: \"7bf04227-6bc5-4797-a189-f1c12841bfc4\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.542487 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.542464 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7bf04227-6bc5-4797-a189-f1c12841bfc4-serving-cert\") pod \"insights-operator-5c847dcfc4-xgcrt\" (UID: \"7bf04227-6bc5-4797-a189-f1c12841bfc4\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.542487 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.542476 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee43e118-575d-4aa1-9276-f9f1ed3606a4-image-registry-private-configuration\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.542625 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.542591 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee43e118-575d-4aa1-9276-f9f1ed3606a4-installation-pull-secrets\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.553248 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.553192 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgwmv\" (UniqueName: \"kubernetes.io/projected/7bf04227-6bc5-4797-a189-f1c12841bfc4-kube-api-access-fgwmv\") pod \"insights-operator-5c847dcfc4-xgcrt\" (UID: \"7bf04227-6bc5-4797-a189-f1c12841bfc4\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.553366 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.553275 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-bound-sa-token\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.553512 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.553491 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrmb8\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-kube-api-access-vrmb8\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:11.638855 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.638774 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-z8zqc\" (UID: \"8aa685e5-fc07-4d7b-bb90-506c9c9f34be\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" Jul 06 19:23:11.638855 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.638813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49nfp\" (UniqueName: \"kubernetes.io/projected/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-kube-api-access-49nfp\") pod \"cluster-monitoring-operator-64786d9cbb-z8zqc\" (UID: \"8aa685e5-fc07-4d7b-bb90-506c9c9f34be\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" Jul 06 19:23:11.638855 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.638830 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07c083da-b946-4438-afa0-3c5be0f77b9b-cert\") pod \"ingress-canary-dk4kn\" (UID: \"07c083da-b946-4438-afa0-3c5be0f77b9b\") " pod="openshift-ingress-canary/ingress-canary-dk4kn" Jul 06 19:23:11.638855 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.638849 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-stats-auth\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:11.639231 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.638867 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66rrx\" (UniqueName: \"kubernetes.io/projected/07c083da-b946-4438-afa0-3c5be0f77b9b-kube-api-access-66rrx\") pod \"ingress-canary-dk4kn\" (UID: \"07c083da-b946-4438-afa0-3c5be0f77b9b\") " pod="openshift-ingress-canary/ingress-canary-dk4kn" Jul 06 19:23:11.639231 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:11.638923 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 19:23:11.639231 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:11.638993 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls podName:8aa685e5-fc07-4d7b-bb90-506c9c9f34be nodeName:}" failed. No retries permitted until 2026-07-06 19:23:12.138972765 +0000 UTC m=+36.767926716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-z8zqc" (UID: "8aa685e5-fc07-4d7b-bb90-506c9c9f34be") : secret "cluster-monitoring-operator-tls" not found Jul 06 19:23:11.639231 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.638926 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-default-certificate\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:11.639231 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.639116 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:11.639231 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.639146 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/670f9fdc-7f0b-44f5-9277-c88477bde835-config-volume\") pod \"dns-default-8j849\" (UID: \"670f9fdc-7f0b-44f5-9277-c88477bde835\") " pod="openshift-dns/dns-default-8j849" Jul 06 19:23:11.639231 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.639181 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/670f9fdc-7f0b-44f5-9277-c88477bde835-tmp-dir\") pod \"dns-default-8j849\" (UID: \"670f9fdc-7f0b-44f5-9277-c88477bde835\") " pod="openshift-dns/dns-default-8j849" Jul 06 19:23:11.639231 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:11.639190 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 19:23:11.639642 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.639247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-z8zqc\" (UID: \"8aa685e5-fc07-4d7b-bb90-506c9c9f34be\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" Jul 06 19:23:11.639642 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.639286 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svwf8\" (UniqueName: \"kubernetes.io/projected/cc56f897-5d59-4451-a517-5bfea063ce0c-kube-api-access-svwf8\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:11.639642 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:11.639342 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs podName:cc56f897-5d59-4451-a517-5bfea063ce0c nodeName:}" failed. No retries permitted until 2026-07-06 19:23:12.139323896 +0000 UTC m=+36.768277854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs") pod "router-default-68d9d48ccc-kk7pz" (UID: "cc56f897-5d59-4451-a517-5bfea063ce0c") : secret "router-metrics-certs-default" not found Jul 06 19:23:11.639642 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.639385 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls\") pod \"dns-default-8j849\" (UID: \"670f9fdc-7f0b-44f5-9277-c88477bde835\") " pod="openshift-dns/dns-default-8j849" Jul 06 19:23:11.639642 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.639410 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q58tp\" (UniqueName: \"kubernetes.io/projected/670f9fdc-7f0b-44f5-9277-c88477bde835-kube-api-access-q58tp\") pod \"dns-default-8j849\" (UID: \"670f9fdc-7f0b-44f5-9277-c88477bde835\") " pod="openshift-dns/dns-default-8j849" Jul 06 19:23:11.639642 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.639442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:11.639642 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:11.639589 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle podName:cc56f897-5d59-4451-a517-5bfea063ce0c nodeName:}" failed. No retries permitted until 2026-07-06 19:23:12.139575768 +0000 UTC m=+36.768529713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle") pod "router-default-68d9d48ccc-kk7pz" (UID: "cc56f897-5d59-4451-a517-5bfea063ce0c") : configmap references non-existent config key: service-ca.crt Jul 06 19:23:11.640063 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.640043 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-z8zqc\" (UID: \"8aa685e5-fc07-4d7b-bb90-506c9c9f34be\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" Jul 06 19:23:11.641400 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.641379 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07c083da-b946-4438-afa0-3c5be0f77b9b-cert\") pod \"ingress-canary-dk4kn\" (UID: \"07c083da-b946-4438-afa0-3c5be0f77b9b\") " pod="openshift-ingress-canary/ingress-canary-dk4kn" Jul 06 19:23:11.641400 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.641387 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-stats-auth\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:11.641581 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.641425 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-default-certificate\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:11.656476 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.656448 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svwf8\" (UniqueName: \"kubernetes.io/projected/cc56f897-5d59-4451-a517-5bfea063ce0c-kube-api-access-svwf8\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:11.656760 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.656738 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49nfp\" (UniqueName: \"kubernetes.io/projected/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-kube-api-access-49nfp\") pod \"cluster-monitoring-operator-64786d9cbb-z8zqc\" (UID: \"8aa685e5-fc07-4d7b-bb90-506c9c9f34be\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" Jul 06 19:23:11.660144 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.660120 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66rrx\" (UniqueName: \"kubernetes.io/projected/07c083da-b946-4438-afa0-3c5be0f77b9b-kube-api-access-66rrx\") pod \"ingress-canary-dk4kn\" (UID: \"07c083da-b946-4438-afa0-3c5be0f77b9b\") " pod="openshift-ingress-canary/ingress-canary-dk4kn" Jul 06 19:23:11.675552 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.675520 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" Jul 06 19:23:11.699085 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.699054 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dk4kn" Jul 06 19:23:11.740561 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.740517 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls\") pod \"dns-default-8j849\" (UID: \"670f9fdc-7f0b-44f5-9277-c88477bde835\") " pod="openshift-dns/dns-default-8j849" Jul 06 19:23:11.740561 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.740569 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58tp\" (UniqueName: \"kubernetes.io/projected/670f9fdc-7f0b-44f5-9277-c88477bde835-kube-api-access-q58tp\") pod \"dns-default-8j849\" (UID: \"670f9fdc-7f0b-44f5-9277-c88477bde835\") " pod="openshift-dns/dns-default-8j849" Jul 06 19:23:11.740822 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:11.740680 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:23:11.740822 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.740744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/670f9fdc-7f0b-44f5-9277-c88477bde835-config-volume\") pod \"dns-default-8j849\" (UID: \"670f9fdc-7f0b-44f5-9277-c88477bde835\") " pod="openshift-dns/dns-default-8j849" Jul 06 19:23:11.740822 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:11.740773 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls podName:670f9fdc-7f0b-44f5-9277-c88477bde835 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:12.240754252 +0000 UTC m=+36.869708200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls") pod "dns-default-8j849" (UID: "670f9fdc-7f0b-44f5-9277-c88477bde835") : secret "dns-default-metrics-tls" not found Jul 06 19:23:11.740822 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.740810 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/670f9fdc-7f0b-44f5-9277-c88477bde835-tmp-dir\") pod \"dns-default-8j849\" (UID: \"670f9fdc-7f0b-44f5-9277-c88477bde835\") " pod="openshift-dns/dns-default-8j849" Jul 06 19:23:11.741337 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.741296 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/670f9fdc-7f0b-44f5-9277-c88477bde835-tmp-dir\") pod \"dns-default-8j849\" (UID: \"670f9fdc-7f0b-44f5-9277-c88477bde835\") " pod="openshift-dns/dns-default-8j849" Jul 06 19:23:11.741478 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.741344 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/670f9fdc-7f0b-44f5-9277-c88477bde835-config-volume\") pod \"dns-default-8j849\" (UID: \"670f9fdc-7f0b-44f5-9277-c88477bde835\") " pod="openshift-dns/dns-default-8j849" Jul 06 19:23:11.750146 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.750093 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q58tp\" (UniqueName: \"kubernetes.io/projected/670f9fdc-7f0b-44f5-9277-c88477bde835-kube-api-access-q58tp\") pod \"dns-default-8j849\" (UID: \"670f9fdc-7f0b-44f5-9277-c88477bde835\") " pod="openshift-dns/dns-default-8j849" Jul 06 19:23:11.794310 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.794024 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-tnbvj"] Jul 06 19:23:11.804874 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.804849 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tnbvj" Jul 06 19:23:11.806906 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.806874 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8tm5l\"" Jul 06 19:23:11.867408 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.867348 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dk4kn"] Jul 06 19:23:11.873657 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.873632 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-xgcrt"] Jul 06 19:23:11.942893 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.942802 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c3ea532-8f74-47ba-bd67-89fc5cd148c3-hosts-file\") pod \"node-resolver-tnbvj\" (UID: \"8c3ea532-8f74-47ba-bd67-89fc5cd148c3\") " pod="openshift-dns/node-resolver-tnbvj" Jul 06 19:23:11.942893 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.942850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqkks\" (UniqueName: \"kubernetes.io/projected/8c3ea532-8f74-47ba-bd67-89fc5cd148c3-kube-api-access-qqkks\") pod \"node-resolver-tnbvj\" (UID: \"8c3ea532-8f74-47ba-bd67-89fc5cd148c3\") " pod="openshift-dns/node-resolver-tnbvj" Jul 06 19:23:11.943116 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:11.942964 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3ea532-8f74-47ba-bd67-89fc5cd148c3-tmp-dir\") pod \"node-resolver-tnbvj\" (UID: \"8c3ea532-8f74-47ba-bd67-89fc5cd148c3\") " pod="openshift-dns/node-resolver-tnbvj" Jul 06 19:23:12.043848 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.043801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c3ea532-8f74-47ba-bd67-89fc5cd148c3-hosts-file\") pod \"node-resolver-tnbvj\" (UID: \"8c3ea532-8f74-47ba-bd67-89fc5cd148c3\") " pod="openshift-dns/node-resolver-tnbvj" Jul 06 19:23:12.043848 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.043854 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqkks\" (UniqueName: \"kubernetes.io/projected/8c3ea532-8f74-47ba-bd67-89fc5cd148c3-kube-api-access-qqkks\") pod \"node-resolver-tnbvj\" (UID: \"8c3ea532-8f74-47ba-bd67-89fc5cd148c3\") " pod="openshift-dns/node-resolver-tnbvj" Jul 06 19:23:12.044112 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.043921 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:12.044112 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.043966 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c3ea532-8f74-47ba-bd67-89fc5cd148c3-hosts-file\") pod \"node-resolver-tnbvj\" (UID: \"8c3ea532-8f74-47ba-bd67-89fc5cd148c3\") " pod="openshift-dns/node-resolver-tnbvj" Jul 06 19:23:12.044112 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.044010 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3ea532-8f74-47ba-bd67-89fc5cd148c3-tmp-dir\") pod \"node-resolver-tnbvj\" (UID: \"8c3ea532-8f74-47ba-bd67-89fc5cd148c3\") " pod="openshift-dns/node-resolver-tnbvj" Jul 06 19:23:12.044246 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:12.044118 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:23:12.044246 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:12.044136 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-954dc4658-xvp8b: secret "image-registry-tls" not found Jul 06 19:23:12.044246 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:12.044195 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls podName:ee43e118-575d-4aa1-9276-f9f1ed3606a4 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:13.044175457 +0000 UTC m=+37.673129421 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls") pod "image-registry-954dc4658-xvp8b" (UID: "ee43e118-575d-4aa1-9276-f9f1ed3606a4") : secret "image-registry-tls" not found Jul 06 19:23:12.044443 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.044423 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3ea532-8f74-47ba-bd67-89fc5cd148c3-tmp-dir\") pod \"node-resolver-tnbvj\" (UID: \"8c3ea532-8f74-47ba-bd67-89fc5cd148c3\") " pod="openshift-dns/node-resolver-tnbvj" Jul 06 19:23:12.052689 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.052664 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqkks\" (UniqueName: \"kubernetes.io/projected/8c3ea532-8f74-47ba-bd67-89fc5cd148c3-kube-api-access-qqkks\") pod \"node-resolver-tnbvj\" (UID: \"8c3ea532-8f74-47ba-bd67-89fc5cd148c3\") " pod="openshift-dns/node-resolver-tnbvj" Jul 06 19:23:12.122891 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.122859 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tnbvj" Jul 06 19:23:12.145282 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.145254 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:12.145436 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.145323 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:12.145436 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.145369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-z8zqc\" (UID: \"8aa685e5-fc07-4d7b-bb90-506c9c9f34be\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" Jul 06 19:23:12.145436 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:12.145407 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle podName:cc56f897-5d59-4451-a517-5bfea063ce0c nodeName:}" failed. No retries permitted until 2026-07-06 19:23:13.145384369 +0000 UTC m=+37.774338327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle") pod "router-default-68d9d48ccc-kk7pz" (UID: "cc56f897-5d59-4451-a517-5bfea063ce0c") : configmap references non-existent config key: service-ca.crt Jul 06 19:23:12.145595 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:12.145432 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 19:23:12.145595 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:12.145456 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 19:23:12.145595 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:12.145495 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs podName:cc56f897-5d59-4451-a517-5bfea063ce0c nodeName:}" failed. No retries permitted until 2026-07-06 19:23:13.14547606 +0000 UTC m=+37.774430009 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs") pod "router-default-68d9d48ccc-kk7pz" (UID: "cc56f897-5d59-4451-a517-5bfea063ce0c") : secret "router-metrics-certs-default" not found Jul 06 19:23:12.145595 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:12.145516 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls podName:8aa685e5-fc07-4d7b-bb90-506c9c9f34be nodeName:}" failed. No retries permitted until 2026-07-06 19:23:13.145505086 +0000 UTC m=+37.774459033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-z8zqc" (UID: "8aa685e5-fc07-4d7b-bb90-506c9c9f34be") : secret "cluster-monitoring-operator-tls" not found Jul 06 19:23:12.246586 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.246498 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls\") pod \"dns-default-8j849\" (UID: \"670f9fdc-7f0b-44f5-9277-c88477bde835\") " pod="openshift-dns/dns-default-8j849" Jul 06 19:23:12.246820 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:12.246667 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:23:12.246820 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:12.246762 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls podName:670f9fdc-7f0b-44f5-9277-c88477bde835 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:13.246740699 +0000 UTC m=+37.875694666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls") pod "dns-default-8j849" (UID: "670f9fdc-7f0b-44f5-9277-c88477bde835") : secret "dns-default-metrics-tls" not found Jul 06 19:23:12.855949 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.855909 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k"] Jul 06 19:23:12.877354 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.877322 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k"] Jul 06 19:23:12.877815 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.877412 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:23:12.877815 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.877422 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:23:12.877815 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.877412 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k" Jul 06 19:23:12.877815 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.877413 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:23:12.886861 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.886837 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 19:23:12.886861 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.886850 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:23:12.887047 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.886902 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d9llp\"" Jul 06 19:23:12.887047 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.886944 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 19:23:12.887047 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.886962 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 06 19:23:12.887213 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.887196 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:23:12.887372 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.887357 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 19:23:12.887465 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.887441 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tr4sh\"" Jul 06 19:23:12.887517 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.887466 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:23:12.887574 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.887449 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:23:12.887574 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.887529 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-2l6qn\"" Jul 06 19:23:12.997267 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:12.997229 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c75b6496c-gpck7"] Jul 06 19:23:13.019350 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.019319 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw"] Jul 06 19:23:13.019519 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.019458 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c75b6496c-gpck7" Jul 06 19:23:13.023262 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.023207 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 06 19:23:13.035439 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.035411 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c75b6496c-gpck7"] Jul 06 19:23:13.035566 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.035535 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.039711 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.039676 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 06 19:23:13.039942 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.039878 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 06 19:23:13.040057 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.040000 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 06 19:23:13.040127 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.040095 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 06 19:23:13.046427 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.046406 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw"] Jul 06 19:23:13.055193 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.055171 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/550ace7a-6e14-4cd8-9470-54099bdac0e3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k\" (UID: \"550ace7a-6e14-4cd8-9470-54099bdac0e3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k" Jul 06 19:23:13.055329 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.055314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z68f8\" (UniqueName: \"kubernetes.io/projected/550ace7a-6e14-4cd8-9470-54099bdac0e3-kube-api-access-z68f8\") pod \"managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k\" (UID: \"550ace7a-6e14-4cd8-9470-54099bdac0e3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k" Jul 06 19:23:13.055447 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.055358 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:13.055518 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:13.055499 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:23:13.055518 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:13.055514 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-954dc4658-xvp8b: secret "image-registry-tls" not found Jul 06 19:23:13.055617 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:13.055563 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls podName:ee43e118-575d-4aa1-9276-f9f1ed3606a4 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:15.055550699 +0000 UTC m=+39.684504644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls") pod "image-registry-954dc4658-xvp8b" (UID: "ee43e118-575d-4aa1-9276-f9f1ed3606a4") : secret "image-registry-tls" not found Jul 06 19:23:13.156421 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.156336 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f101d66-483d-43f5-89f7-cb8dcc39f131-additional-service-ca\") pod \"cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw\" (UID: \"0f101d66-483d-43f5-89f7-cb8dcc39f131\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.156580 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.156446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0f101d66-483d-43f5-89f7-cb8dcc39f131-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw\" (UID: \"0f101d66-483d-43f5-89f7-cb8dcc39f131\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.156580 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.156488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:13.156580 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.156527 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0f101d66-483d-43f5-89f7-cb8dcc39f131-ca\") pod \"cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw\" (UID: \"0f101d66-483d-43f5-89f7-cb8dcc39f131\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.156580 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.156552 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdxtv\" (UniqueName: \"kubernetes.io/projected/0f101d66-483d-43f5-89f7-cb8dcc39f131-kube-api-access-bdxtv\") pod \"cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw\" (UID: \"0f101d66-483d-43f5-89f7-cb8dcc39f131\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.156794 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.156582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z68f8\" (UniqueName: \"kubernetes.io/projected/550ace7a-6e14-4cd8-9470-54099bdac0e3-kube-api-access-z68f8\") pod \"managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k\" (UID: \"550ace7a-6e14-4cd8-9470-54099bdac0e3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k" Jul 06 19:23:13.156794 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:13.156630 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle podName:cc56f897-5d59-4451-a517-5bfea063ce0c nodeName:}" failed. No retries permitted until 2026-07-06 19:23:15.156611887 +0000 UTC m=+39.785565835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle") pod "router-default-68d9d48ccc-kk7pz" (UID: "cc56f897-5d59-4451-a517-5bfea063ce0c") : configmap references non-existent config key: service-ca.crt Jul 06 19:23:13.156794 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.156687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:13.156794 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.156741 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/586de6b0-7270-4ddf-b643-d89d0d21e61e-tmp\") pod \"klusterlet-addon-workmgr-6c75b6496c-gpck7\" (UID: \"586de6b0-7270-4ddf-b643-d89d0d21e61e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c75b6496c-gpck7" Jul 06 19:23:13.156794 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.156763 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/586de6b0-7270-4ddf-b643-d89d0d21e61e-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c75b6496c-gpck7\" (UID: \"586de6b0-7270-4ddf-b643-d89d0d21e61e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c75b6496c-gpck7" Jul 06 19:23:13.157026 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.156833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-z8zqc\" (UID: \"8aa685e5-fc07-4d7b-bb90-506c9c9f34be\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" Jul 06 19:23:13.157026 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:13.156855 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 19:23:13.157026 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.156888 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/550ace7a-6e14-4cd8-9470-54099bdac0e3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k\" (UID: \"550ace7a-6e14-4cd8-9470-54099bdac0e3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k" Jul 06 19:23:13.157026 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:13.156917 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs podName:cc56f897-5d59-4451-a517-5bfea063ce0c nodeName:}" failed. No retries permitted until 2026-07-06 19:23:15.156903514 +0000 UTC m=+39.785857461 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs") pod "router-default-68d9d48ccc-kk7pz" (UID: "cc56f897-5d59-4451-a517-5bfea063ce0c") : secret "router-metrics-certs-default" not found Jul 06 19:23:13.157026 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:13.156953 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 19:23:13.157026 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:13.157018 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls podName:8aa685e5-fc07-4d7b-bb90-506c9c9f34be nodeName:}" failed. No retries permitted until 2026-07-06 19:23:15.157000234 +0000 UTC m=+39.785954192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-z8zqc" (UID: "8aa685e5-fc07-4d7b-bb90-506c9c9f34be") : secret "cluster-monitoring-operator-tls" not found Jul 06 19:23:13.157026 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.156954 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f101d66-483d-43f5-89f7-cb8dcc39f131-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw\" (UID: \"0f101d66-483d-43f5-89f7-cb8dcc39f131\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.157392 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.157065 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0f101d66-483d-43f5-89f7-cb8dcc39f131-hub\") pod \"cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw\" (UID: \"0f101d66-483d-43f5-89f7-cb8dcc39f131\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.157392 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.157102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl5qk\" (UniqueName: \"kubernetes.io/projected/586de6b0-7270-4ddf-b643-d89d0d21e61e-kube-api-access-cl5qk\") pod \"klusterlet-addon-workmgr-6c75b6496c-gpck7\" (UID: \"586de6b0-7270-4ddf-b643-d89d0d21e61e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c75b6496c-gpck7" Jul 06 19:23:13.159627 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.159603 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/550ace7a-6e14-4cd8-9470-54099bdac0e3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k\" (UID: \"550ace7a-6e14-4cd8-9470-54099bdac0e3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k" Jul 06 19:23:13.166057 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.166026 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z68f8\" (UniqueName: \"kubernetes.io/projected/550ace7a-6e14-4cd8-9470-54099bdac0e3-kube-api-access-z68f8\") pod \"managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k\" (UID: \"550ace7a-6e14-4cd8-9470-54099bdac0e3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k" Jul 06 19:23:13.216910 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.216874 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k" Jul 06 19:23:13.257538 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.257508 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f101d66-483d-43f5-89f7-cb8dcc39f131-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw\" (UID: \"0f101d66-483d-43f5-89f7-cb8dcc39f131\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.257725 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.257545 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0f101d66-483d-43f5-89f7-cb8dcc39f131-hub\") pod \"cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw\" (UID: \"0f101d66-483d-43f5-89f7-cb8dcc39f131\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.257725 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.257574 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl5qk\" (UniqueName: \"kubernetes.io/projected/586de6b0-7270-4ddf-b643-d89d0d21e61e-kube-api-access-cl5qk\") pod \"klusterlet-addon-workmgr-6c75b6496c-gpck7\" (UID: \"586de6b0-7270-4ddf-b643-d89d0d21e61e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c75b6496c-gpck7" Jul 06 19:23:13.257725 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.257625 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f101d66-483d-43f5-89f7-cb8dcc39f131-additional-service-ca\") pod \"cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw\" (UID: \"0f101d66-483d-43f5-89f7-cb8dcc39f131\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.257725 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.257661 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls\") pod \"dns-default-8j849\" (UID: \"670f9fdc-7f0b-44f5-9277-c88477bde835\") " pod="openshift-dns/dns-default-8j849" Jul 06 19:23:13.257725 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.257689 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0f101d66-483d-43f5-89f7-cb8dcc39f131-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw\" (UID: \"0f101d66-483d-43f5-89f7-cb8dcc39f131\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.257974 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.257756 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0f101d66-483d-43f5-89f7-cb8dcc39f131-ca\") pod \"cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw\" (UID: \"0f101d66-483d-43f5-89f7-cb8dcc39f131\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.257974 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.257782 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdxtv\" (UniqueName: \"kubernetes.io/projected/0f101d66-483d-43f5-89f7-cb8dcc39f131-kube-api-access-bdxtv\") pod \"cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw\" (UID: \"0f101d66-483d-43f5-89f7-cb8dcc39f131\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.257974 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.257874 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/586de6b0-7270-4ddf-b643-d89d0d21e61e-tmp\") pod \"klusterlet-addon-workmgr-6c75b6496c-gpck7\" (UID: \"586de6b0-7270-4ddf-b643-d89d0d21e61e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c75b6496c-gpck7" Jul 06 19:23:13.257974 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.257906 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/586de6b0-7270-4ddf-b643-d89d0d21e61e-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c75b6496c-gpck7\" (UID: \"586de6b0-7270-4ddf-b643-d89d0d21e61e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c75b6496c-gpck7" Jul 06 19:23:13.258936 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:13.258315 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:23:13.258936 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:13.258377 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls podName:670f9fdc-7f0b-44f5-9277-c88477bde835 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:15.258356931 +0000 UTC m=+39.887310880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls") pod "dns-default-8j849" (UID: "670f9fdc-7f0b-44f5-9277-c88477bde835") : secret "dns-default-metrics-tls" not found Jul 06 19:23:13.258936 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.258882 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f101d66-483d-43f5-89f7-cb8dcc39f131-additional-service-ca\") pod \"cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw\" (UID: \"0f101d66-483d-43f5-89f7-cb8dcc39f131\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.259358 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.259316 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/586de6b0-7270-4ddf-b643-d89d0d21e61e-tmp\") pod \"klusterlet-addon-workmgr-6c75b6496c-gpck7\" (UID: \"586de6b0-7270-4ddf-b643-d89d0d21e61e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c75b6496c-gpck7" Jul 06 19:23:13.260840 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.260817 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/586de6b0-7270-4ddf-b643-d89d0d21e61e-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c75b6496c-gpck7\" (UID: \"586de6b0-7270-4ddf-b643-d89d0d21e61e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c75b6496c-gpck7" Jul 06 19:23:13.261011 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.260991 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0f101d66-483d-43f5-89f7-cb8dcc39f131-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw\" (UID: \"0f101d66-483d-43f5-89f7-cb8dcc39f131\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.261169 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.261139 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0f101d66-483d-43f5-89f7-cb8dcc39f131-hub\") pod \"cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw\" (UID: \"0f101d66-483d-43f5-89f7-cb8dcc39f131\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.261326 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.261305 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f101d66-483d-43f5-89f7-cb8dcc39f131-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw\" (UID: \"0f101d66-483d-43f5-89f7-cb8dcc39f131\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.261573 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.261557 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0f101d66-483d-43f5-89f7-cb8dcc39f131-ca\") pod \"cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw\" (UID: \"0f101d66-483d-43f5-89f7-cb8dcc39f131\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.267548 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.267523 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl5qk\" (UniqueName: \"kubernetes.io/projected/586de6b0-7270-4ddf-b643-d89d0d21e61e-kube-api-access-cl5qk\") pod \"klusterlet-addon-workmgr-6c75b6496c-gpck7\" (UID: \"586de6b0-7270-4ddf-b643-d89d0d21e61e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c75b6496c-gpck7" Jul 06 19:23:13.268116 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.268083 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdxtv\" (UniqueName: \"kubernetes.io/projected/0f101d66-483d-43f5-89f7-cb8dcc39f131-kube-api-access-bdxtv\") pod \"cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw\" (UID: \"0f101d66-483d-43f5-89f7-cb8dcc39f131\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.330500 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.330462 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c75b6496c-gpck7" Jul 06 19:23:13.346564 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.346535 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:23:13.369782 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:13.369746 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf04227_6bc5_4797_a189_f1c12841bfc4.slice/crio-8b548b0e4ca2994bbfac0b9d5a64e42f1506a7a53cbbfc4aba3eb0aadf4dd6cb WatchSource:0}: Error finding container 8b548b0e4ca2994bbfac0b9d5a64e42f1506a7a53cbbfc4aba3eb0aadf4dd6cb: Status 404 returned error can't find the container with id 8b548b0e4ca2994bbfac0b9d5a64e42f1506a7a53cbbfc4aba3eb0aadf4dd6cb Jul 06 19:23:13.370798 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:13.370769 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c083da_b946_4438_afa0_3c5be0f77b9b.slice/crio-9a78f0a0b8959dfedc7cfc0235c85ec498b23b945f3700a541b39c1c2e50e846 WatchSource:0}: Error finding container 9a78f0a0b8959dfedc7cfc0235c85ec498b23b945f3700a541b39c1c2e50e846: Status 404 returned error can't find the container with id 9a78f0a0b8959dfedc7cfc0235c85ec498b23b945f3700a541b39c1c2e50e846 Jul 06 19:23:13.381190 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:13.381160 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3ea532_8f74_47ba_bd67_89fc5cd148c3.slice/crio-3c71e87ca33c484765d5c4491a1e796bb2b62b5404ab1f5e0e8aef3b7a681fc8 WatchSource:0}: Error finding container 3c71e87ca33c484765d5c4491a1e796bb2b62b5404ab1f5e0e8aef3b7a681fc8: Status 404 returned error can't find the container with id 3c71e87ca33c484765d5c4491a1e796bb2b62b5404ab1f5e0e8aef3b7a681fc8 Jul 06 19:23:13.552854 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.552625 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k"] Jul 06 19:23:13.567212 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.567182 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c75b6496c-gpck7"] Jul 06 19:23:13.574638 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:13.574605 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw"] Jul 06 19:23:13.678256 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:13.678136 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod550ace7a_6e14_4cd8_9470_54099bdac0e3.slice/crio-9baaa14d101f2859987a77af48989700de602d85c6fd0ae695a25caff491fac6 WatchSource:0}: Error finding container 9baaa14d101f2859987a77af48989700de602d85c6fd0ae695a25caff491fac6: Status 404 returned error can't find the container with id 9baaa14d101f2859987a77af48989700de602d85c6fd0ae695a25caff491fac6 Jul 06 19:23:13.678429 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:13.678390 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586de6b0_7270_4ddf_b643_d89d0d21e61e.slice/crio-4990d73daccd5f034ec7f4a83b6ae8cade3cbdc707331625a27b26b857c9f03e WatchSource:0}: Error finding container 4990d73daccd5f034ec7f4a83b6ae8cade3cbdc707331625a27b26b857c9f03e: Status 404 returned error can't find the container with id 4990d73daccd5f034ec7f4a83b6ae8cade3cbdc707331625a27b26b857c9f03e Jul 06 19:23:13.679297 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:13.679200 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f101d66_483d_43f5_89f7_cb8dcc39f131.slice/crio-d7bcf355a66649970714c4e4ea5abc7bfbe0effc7153374fac07057327a19453 WatchSource:0}: Error finding container d7bcf355a66649970714c4e4ea5abc7bfbe0effc7153374fac07057327a19453: Status 404 returned error can't find the container with id d7bcf355a66649970714c4e4ea5abc7bfbe0effc7153374fac07057327a19453 Jul 06 19:23:14.033229 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:14.033195 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k" event={"ID":"550ace7a-6e14-4cd8-9470-54099bdac0e3","Type":"ContainerStarted","Data":"9baaa14d101f2859987a77af48989700de602d85c6fd0ae695a25caff491fac6"} Jul 06 19:23:14.034424 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:14.034389 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dk4kn" event={"ID":"07c083da-b946-4438-afa0-3c5be0f77b9b","Type":"ContainerStarted","Data":"9a78f0a0b8959dfedc7cfc0235c85ec498b23b945f3700a541b39c1c2e50e846"} Jul 06 19:23:14.035582 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:14.035557 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c75b6496c-gpck7" event={"ID":"586de6b0-7270-4ddf-b643-d89d0d21e61e","Type":"ContainerStarted","Data":"4990d73daccd5f034ec7f4a83b6ae8cade3cbdc707331625a27b26b857c9f03e"} Jul 06 19:23:14.036759 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:14.036732 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" event={"ID":"0f101d66-483d-43f5-89f7-cb8dcc39f131","Type":"ContainerStarted","Data":"d7bcf355a66649970714c4e4ea5abc7bfbe0effc7153374fac07057327a19453"} Jul 06 19:23:14.038163 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:14.038143 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tnbvj" event={"ID":"8c3ea532-8f74-47ba-bd67-89fc5cd148c3","Type":"ContainerStarted","Data":"58d52e4d3ce7ebd75e27e090cc1ec2f4655b6798a8cb3d5ffae8400b3e91d118"} Jul 06 19:23:14.038271 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:14.038188 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tnbvj" event={"ID":"8c3ea532-8f74-47ba-bd67-89fc5cd148c3","Type":"ContainerStarted","Data":"3c71e87ca33c484765d5c4491a1e796bb2b62b5404ab1f5e0e8aef3b7a681fc8"} Jul 06 19:23:14.039366 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:14.039342 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" event={"ID":"7bf04227-6bc5-4797-a189-f1c12841bfc4","Type":"ContainerStarted","Data":"8b548b0e4ca2994bbfac0b9d5a64e42f1506a7a53cbbfc4aba3eb0aadf4dd6cb"} Jul 06 19:23:14.042520 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:14.042495 2578 generic.go:356] "Generic (PLEG): container finished" podID="3e4c4a93-9461-4a11-aef0-e1399f9bc611" containerID="9b54f3b88555fabfdfa49d7f773d98ec66793bbb2b350a29a68fea170e63d43a" exitCode=0 Jul 06 19:23:14.042623 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:14.042533 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wk4zk" event={"ID":"3e4c4a93-9461-4a11-aef0-e1399f9bc611","Type":"ContainerDied","Data":"9b54f3b88555fabfdfa49d7f773d98ec66793bbb2b350a29a68fea170e63d43a"} Jul 06 19:23:14.055553 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:14.055506 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tnbvj" podStartSLOduration=3.055492057 podStartE2EDuration="3.055492057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:14.054458816 +0000 UTC m=+38.683412783" watchObservedRunningTime="2026-07-06 19:23:14.055492057 +0000 UTC m=+38.684446022" Jul 06 19:23:15.071564 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:15.071339 2578 generic.go:356] "Generic (PLEG): container finished" podID="3e4c4a93-9461-4a11-aef0-e1399f9bc611" containerID="b1b6f9714a0e1e679fd885b1972d2b5fb35a8d705f6f129a407865f7be10d1ff" exitCode=0 Jul 06 19:23:15.072044 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:15.071632 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wk4zk" event={"ID":"3e4c4a93-9461-4a11-aef0-e1399f9bc611","Type":"ContainerDied","Data":"b1b6f9714a0e1e679fd885b1972d2b5fb35a8d705f6f129a407865f7be10d1ff"} Jul 06 19:23:15.077197 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:15.076353 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:15.077197 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:15.076579 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:23:15.077197 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:15.076596 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-954dc4658-xvp8b: secret "image-registry-tls" not found Jul 06 19:23:15.077197 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:15.076649 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls podName:ee43e118-575d-4aa1-9276-f9f1ed3606a4 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:19.076630052 +0000 UTC m=+43.705583997 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls") pod "image-registry-954dc4658-xvp8b" (UID: "ee43e118-575d-4aa1-9276-f9f1ed3606a4") : secret "image-registry-tls" not found Jul 06 19:23:15.177597 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:15.177555 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:15.177790 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:15.177675 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-z8zqc\" (UID: \"8aa685e5-fc07-4d7b-bb90-506c9c9f34be\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" Jul 06 19:23:15.177886 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:15.177792 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:15.178200 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:15.178177 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 19:23:15.178286 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:15.178245 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs podName:cc56f897-5d59-4451-a517-5bfea063ce0c nodeName:}" failed. No retries permitted until 2026-07-06 19:23:19.178225002 +0000 UTC m=+43.807178963 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs") pod "router-default-68d9d48ccc-kk7pz" (UID: "cc56f897-5d59-4451-a517-5bfea063ce0c") : secret "router-metrics-certs-default" not found Jul 06 19:23:15.179013 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:15.178992 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 19:23:15.179113 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:15.179049 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls podName:8aa685e5-fc07-4d7b-bb90-506c9c9f34be nodeName:}" failed. No retries permitted until 2026-07-06 19:23:19.179033016 +0000 UTC m=+43.807986975 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-z8zqc" (UID: "8aa685e5-fc07-4d7b-bb90-506c9c9f34be") : secret "cluster-monitoring-operator-tls" not found Jul 06 19:23:15.179503 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:15.179486 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle podName:cc56f897-5d59-4451-a517-5bfea063ce0c nodeName:}" failed. No retries permitted until 2026-07-06 19:23:19.179471218 +0000 UTC m=+43.808425173 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle") pod "router-default-68d9d48ccc-kk7pz" (UID: "cc56f897-5d59-4451-a517-5bfea063ce0c") : configmap references non-existent config key: service-ca.crt Jul 06 19:23:15.279794 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:15.279168 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls\") pod \"dns-default-8j849\" (UID: \"670f9fdc-7f0b-44f5-9277-c88477bde835\") " pod="openshift-dns/dns-default-8j849" Jul 06 19:23:15.279794 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:15.279333 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:23:15.279794 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:15.279402 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls podName:670f9fdc-7f0b-44f5-9277-c88477bde835 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:19.279383242 +0000 UTC m=+43.908337191 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls") pod "dns-default-8j849" (UID: "670f9fdc-7f0b-44f5-9277-c88477bde835") : secret "dns-default-metrics-tls" not found Jul 06 19:23:16.081559 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:16.081519 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wk4zk" event={"ID":"3e4c4a93-9461-4a11-aef0-e1399f9bc611","Type":"ContainerStarted","Data":"5df754f285c3236948e0a55ec8ee260567f97fed86071136da6fe2ee7981e6cc"} Jul 06 19:23:16.111479 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:16.111095 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wk4zk" podStartSLOduration=5.957426255 podStartE2EDuration="40.111053551s" totalImagesPullingTime="34.153627296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:16.109434994 +0000 UTC m=+40.738388963" watchObservedRunningTime="2026-07-06 19:23:16.111053551 +0000 UTC m=+40.740007518" Jul 06 19:23:19.119278 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:19.119232 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:19.119721 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:19.119381 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:23:19.119721 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:19.119398 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-954dc4658-xvp8b: secret "image-registry-tls" not found Jul 06 19:23:19.119721 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:19.119462 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls podName:ee43e118-575d-4aa1-9276-f9f1ed3606a4 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:27.119445948 +0000 UTC m=+51.748399892 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls") pod "image-registry-954dc4658-xvp8b" (UID: "ee43e118-575d-4aa1-9276-f9f1ed3606a4") : secret "image-registry-tls" not found Jul 06 19:23:19.220715 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:19.220657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:19.220867 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:19.220745 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-z8zqc\" (UID: \"8aa685e5-fc07-4d7b-bb90-506c9c9f34be\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" Jul 06 19:23:19.220867 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:19.220813 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 19:23:19.220987 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:19.220894 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs podName:cc56f897-5d59-4451-a517-5bfea063ce0c nodeName:}" failed. No retries permitted until 2026-07-06 19:23:27.220870403 +0000 UTC m=+51.849824358 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs") pod "router-default-68d9d48ccc-kk7pz" (UID: "cc56f897-5d59-4451-a517-5bfea063ce0c") : secret "router-metrics-certs-default" not found Jul 06 19:23:19.220987 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:19.220917 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle podName:cc56f897-5d59-4451-a517-5bfea063ce0c nodeName:}" failed. No retries permitted until 2026-07-06 19:23:27.220904805 +0000 UTC m=+51.849858748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle") pod "router-default-68d9d48ccc-kk7pz" (UID: "cc56f897-5d59-4451-a517-5bfea063ce0c") : configmap references non-existent config key: service-ca.crt Jul 06 19:23:19.220987 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:19.220966 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 19:23:19.221153 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:19.220991 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls podName:8aa685e5-fc07-4d7b-bb90-506c9c9f34be nodeName:}" failed. No retries permitted until 2026-07-06 19:23:27.220983868 +0000 UTC m=+51.849937812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-z8zqc" (UID: "8aa685e5-fc07-4d7b-bb90-506c9c9f34be") : secret "cluster-monitoring-operator-tls" not found Jul 06 19:23:19.221153 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:19.220822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:19.322319 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:19.322280 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls\") pod \"dns-default-8j849\" (UID: \"670f9fdc-7f0b-44f5-9277-c88477bde835\") " pod="openshift-dns/dns-default-8j849" Jul 06 19:23:19.322497 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:19.322450 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:23:19.322614 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:19.322600 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls podName:670f9fdc-7f0b-44f5-9277-c88477bde835 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:27.322579357 +0000 UTC m=+51.951533305 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls") pod "dns-default-8j849" (UID: "670f9fdc-7f0b-44f5-9277-c88477bde835") : secret "dns-default-metrics-tls" not found Jul 06 19:23:19.702228 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:19.702188 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph"] Jul 06 19:23:19.734320 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:19.734278 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph"] Jul 06 19:23:19.734497 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:19.734407 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph" Jul 06 19:23:19.736886 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:19.736854 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5njvv\"" Jul 06 19:23:19.737378 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:19.737353 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 06 19:23:19.737487 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:19.737405 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 06 19:23:19.737563 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:19.737355 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:23:19.826906 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:19.826870 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c76c3d74-82a5-4c4c-9aab-876dd450bd83-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2gvph\" (UID: \"c76c3d74-82a5-4c4c-9aab-876dd450bd83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph" Jul 06 19:23:19.827131 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:19.827108 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg2sn\" (UniqueName: \"kubernetes.io/projected/c76c3d74-82a5-4c4c-9aab-876dd450bd83-kube-api-access-dg2sn\") pod \"cluster-samples-operator-56fcf86bf-2gvph\" (UID: \"c76c3d74-82a5-4c4c-9aab-876dd450bd83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph" Jul 06 19:23:19.928044 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:19.928004 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg2sn\" (UniqueName: \"kubernetes.io/projected/c76c3d74-82a5-4c4c-9aab-876dd450bd83-kube-api-access-dg2sn\") pod \"cluster-samples-operator-56fcf86bf-2gvph\" (UID: \"c76c3d74-82a5-4c4c-9aab-876dd450bd83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph" Jul 06 19:23:19.928044 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:19.928057 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c76c3d74-82a5-4c4c-9aab-876dd450bd83-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2gvph\" (UID: \"c76c3d74-82a5-4c4c-9aab-876dd450bd83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph" Jul 06 19:23:19.928288 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:19.928202 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:23:19.928288 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:19.928270 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c76c3d74-82a5-4c4c-9aab-876dd450bd83-samples-operator-tls podName:c76c3d74-82a5-4c4c-9aab-876dd450bd83 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:20.428253766 +0000 UTC m=+45.057207715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c76c3d74-82a5-4c4c-9aab-876dd450bd83-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2gvph" (UID: "c76c3d74-82a5-4c4c-9aab-876dd450bd83") : secret "samples-operator-tls" not found Jul 06 19:23:19.941137 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:19.941106 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg2sn\" (UniqueName: \"kubernetes.io/projected/c76c3d74-82a5-4c4c-9aab-876dd450bd83-kube-api-access-dg2sn\") pod \"cluster-samples-operator-56fcf86bf-2gvph\" (UID: \"c76c3d74-82a5-4c4c-9aab-876dd450bd83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph" Jul 06 19:23:20.432863 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.432819 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c76c3d74-82a5-4c4c-9aab-876dd450bd83-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2gvph\" (UID: \"c76c3d74-82a5-4c4c-9aab-876dd450bd83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph" Jul 06 19:23:20.433238 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:20.432982 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:23:20.433238 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:20.433049 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c76c3d74-82a5-4c4c-9aab-876dd450bd83-samples-operator-tls podName:c76c3d74-82a5-4c4c-9aab-876dd450bd83 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:21.433032552 +0000 UTC m=+46.061986505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c76c3d74-82a5-4c4c-9aab-876dd450bd83-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2gvph" (UID: "c76c3d74-82a5-4c4c-9aab-876dd450bd83") : secret "samples-operator-tls" not found Jul 06 19:23:20.755506 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.755433 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dqgwt"] Jul 06 19:23:20.778437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.778401 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-bx5pn"] Jul 06 19:23:20.779580 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.779078 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dqgwt" Jul 06 19:23:20.782765 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.782742 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:23:20.782999 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.782786 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-bwnwk\"" Jul 06 19:23:20.783125 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.782842 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 06 19:23:20.795342 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.795316 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-62lvt"] Jul 06 19:23:20.795440 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.795350 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-bx5pn" Jul 06 19:23:20.797669 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.797646 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-zj72b\"" Jul 06 19:23:20.797862 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.797847 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:23:20.798181 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.798150 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 06 19:23:20.798339 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.798323 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 06 19:23:20.798339 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.798337 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 06 19:23:20.816544 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.816522 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dqgwt"] Jul 06 19:23:20.816676 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.816557 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-bx5pn"] Jul 06 19:23:20.816676 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.816577 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-62lvt"] Jul 06 19:23:20.816676 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.816636 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-62lvt" Jul 06 19:23:20.819625 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.819604 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 06 19:23:20.819762 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.819555 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 06 19:23:20.819762 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.819652 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 06 19:23:20.820109 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.820062 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dmhhb\"" Jul 06 19:23:20.821795 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.821600 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:23:20.827834 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.827812 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 06 19:23:20.836585 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.836564 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fccf3d5c-efcb-456c-8dc9-9aa7191184d4-config\") pod \"service-ca-operator-5d46d9ffd4-bx5pn\" (UID: \"fccf3d5c-efcb-456c-8dc9-9aa7191184d4\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-bx5pn" Jul 06 19:23:20.836685 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.836598 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fccf3d5c-efcb-456c-8dc9-9aa7191184d4-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-bx5pn\" (UID: \"fccf3d5c-efcb-456c-8dc9-9aa7191184d4\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-bx5pn" Jul 06 19:23:20.836685 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.836623 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09143401-93b0-453e-a738-9b8fa5457a81-trusted-ca\") pod \"console-operator-68b757fb67-62lvt\" (UID: \"09143401-93b0-453e-a738-9b8fa5457a81\") " pod="openshift-console-operator/console-operator-68b757fb67-62lvt" Jul 06 19:23:20.836685 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.836675 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09143401-93b0-453e-a738-9b8fa5457a81-serving-cert\") pod \"console-operator-68b757fb67-62lvt\" (UID: \"09143401-93b0-453e-a738-9b8fa5457a81\") " pod="openshift-console-operator/console-operator-68b757fb67-62lvt" Jul 06 19:23:20.836874 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.836754 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28h5q\" (UniqueName: \"kubernetes.io/projected/09143401-93b0-453e-a738-9b8fa5457a81-kube-api-access-28h5q\") pod \"console-operator-68b757fb67-62lvt\" (UID: \"09143401-93b0-453e-a738-9b8fa5457a81\") " pod="openshift-console-operator/console-operator-68b757fb67-62lvt" Jul 06 19:23:20.836874 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.836784 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbcqc\" (UniqueName: \"kubernetes.io/projected/fccf3d5c-efcb-456c-8dc9-9aa7191184d4-kube-api-access-zbcqc\") pod \"service-ca-operator-5d46d9ffd4-bx5pn\" (UID: \"fccf3d5c-efcb-456c-8dc9-9aa7191184d4\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-bx5pn" Jul 06 19:23:20.836874 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.836821 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbflh\" (UniqueName: \"kubernetes.io/projected/f9180d35-c91f-42b4-b49d-eebdf8b3a1c3-kube-api-access-wbflh\") pod \"volume-data-source-validator-5d456fdbcd-dqgwt\" (UID: \"f9180d35-c91f-42b4-b49d-eebdf8b3a1c3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dqgwt" Jul 06 19:23:20.837035 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.836952 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09143401-93b0-453e-a738-9b8fa5457a81-config\") pod \"console-operator-68b757fb67-62lvt\" (UID: \"09143401-93b0-453e-a738-9b8fa5457a81\") " pod="openshift-console-operator/console-operator-68b757fb67-62lvt" Jul 06 19:23:20.860854 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.860814 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-rhbbf"] Jul 06 19:23:20.882589 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.882554 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-rhbbf"] Jul 06 19:23:20.882774 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.882708 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-rhbbf" Jul 06 19:23:20.886491 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.886469 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:23:20.886491 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.886488 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 06 19:23:20.886673 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.886500 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 06 19:23:20.886931 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.886914 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 06 19:23:20.887160 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.887000 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-q456w\"" Jul 06 19:23:20.938493 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.938456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09143401-93b0-453e-a738-9b8fa5457a81-config\") pod \"console-operator-68b757fb67-62lvt\" (UID: \"09143401-93b0-453e-a738-9b8fa5457a81\") " pod="openshift-console-operator/console-operator-68b757fb67-62lvt" Jul 06 19:23:20.938715 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.938525 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fccf3d5c-efcb-456c-8dc9-9aa7191184d4-config\") pod \"service-ca-operator-5d46d9ffd4-bx5pn\" (UID: \"fccf3d5c-efcb-456c-8dc9-9aa7191184d4\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-bx5pn" Jul 06 19:23:20.938715 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.938554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fccf3d5c-efcb-456c-8dc9-9aa7191184d4-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-bx5pn\" (UID: \"fccf3d5c-efcb-456c-8dc9-9aa7191184d4\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-bx5pn" Jul 06 19:23:20.938715 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.938578 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09143401-93b0-453e-a738-9b8fa5457a81-trusted-ca\") pod \"console-operator-68b757fb67-62lvt\" (UID: \"09143401-93b0-453e-a738-9b8fa5457a81\") " pod="openshift-console-operator/console-operator-68b757fb67-62lvt" Jul 06 19:23:20.938715 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.938607 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f02ed85e-784b-4f19-bd40-c79ebd77a8a9-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-rhbbf\" (UID: \"f02ed85e-784b-4f19-bd40-c79ebd77a8a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-rhbbf" Jul 06 19:23:20.938715 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.938636 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnfv5\" (UniqueName: \"kubernetes.io/projected/f02ed85e-784b-4f19-bd40-c79ebd77a8a9-kube-api-access-vnfv5\") pod \"kube-storage-version-migrator-operator-846bf668f7-rhbbf\" (UID: \"f02ed85e-784b-4f19-bd40-c79ebd77a8a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-rhbbf" Jul 06 19:23:20.938715 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.938713 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f02ed85e-784b-4f19-bd40-c79ebd77a8a9-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-rhbbf\" (UID: \"f02ed85e-784b-4f19-bd40-c79ebd77a8a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-rhbbf" Jul 06 19:23:20.939025 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.938743 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09143401-93b0-453e-a738-9b8fa5457a81-serving-cert\") pod \"console-operator-68b757fb67-62lvt\" (UID: \"09143401-93b0-453e-a738-9b8fa5457a81\") " pod="openshift-console-operator/console-operator-68b757fb67-62lvt" Jul 06 19:23:20.939025 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.938799 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28h5q\" (UniqueName: \"kubernetes.io/projected/09143401-93b0-453e-a738-9b8fa5457a81-kube-api-access-28h5q\") pod \"console-operator-68b757fb67-62lvt\" (UID: \"09143401-93b0-453e-a738-9b8fa5457a81\") " pod="openshift-console-operator/console-operator-68b757fb67-62lvt" Jul 06 19:23:20.939025 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.938881 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbcqc\" (UniqueName: \"kubernetes.io/projected/fccf3d5c-efcb-456c-8dc9-9aa7191184d4-kube-api-access-zbcqc\") pod \"service-ca-operator-5d46d9ffd4-bx5pn\" (UID: \"fccf3d5c-efcb-456c-8dc9-9aa7191184d4\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-bx5pn" Jul 06 19:23:20.939025 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.938930 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbflh\" (UniqueName: \"kubernetes.io/projected/f9180d35-c91f-42b4-b49d-eebdf8b3a1c3-kube-api-access-wbflh\") pod \"volume-data-source-validator-5d456fdbcd-dqgwt\" (UID: \"f9180d35-c91f-42b4-b49d-eebdf8b3a1c3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dqgwt" Jul 06 19:23:20.939227 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.939165 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fccf3d5c-efcb-456c-8dc9-9aa7191184d4-config\") pod \"service-ca-operator-5d46d9ffd4-bx5pn\" (UID: \"fccf3d5c-efcb-456c-8dc9-9aa7191184d4\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-bx5pn" Jul 06 19:23:20.939345 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.939315 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09143401-93b0-453e-a738-9b8fa5457a81-config\") pod \"console-operator-68b757fb67-62lvt\" (UID: \"09143401-93b0-453e-a738-9b8fa5457a81\") " pod="openshift-console-operator/console-operator-68b757fb67-62lvt" Jul 06 19:23:20.939973 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.939955 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09143401-93b0-453e-a738-9b8fa5457a81-trusted-ca\") pod \"console-operator-68b757fb67-62lvt\" (UID: \"09143401-93b0-453e-a738-9b8fa5457a81\") " pod="openshift-console-operator/console-operator-68b757fb67-62lvt" Jul 06 19:23:20.941493 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.941458 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09143401-93b0-453e-a738-9b8fa5457a81-serving-cert\") pod \"console-operator-68b757fb67-62lvt\" (UID: \"09143401-93b0-453e-a738-9b8fa5457a81\") " pod="openshift-console-operator/console-operator-68b757fb67-62lvt" Jul 06 19:23:20.941493 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.941482 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fccf3d5c-efcb-456c-8dc9-9aa7191184d4-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-bx5pn\" (UID: \"fccf3d5c-efcb-456c-8dc9-9aa7191184d4\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-bx5pn" Jul 06 19:23:20.950938 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.950861 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbcqc\" (UniqueName: \"kubernetes.io/projected/fccf3d5c-efcb-456c-8dc9-9aa7191184d4-kube-api-access-zbcqc\") pod \"service-ca-operator-5d46d9ffd4-bx5pn\" (UID: \"fccf3d5c-efcb-456c-8dc9-9aa7191184d4\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-bx5pn" Jul 06 19:23:20.950938 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.950866 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28h5q\" (UniqueName: \"kubernetes.io/projected/09143401-93b0-453e-a738-9b8fa5457a81-kube-api-access-28h5q\") pod \"console-operator-68b757fb67-62lvt\" (UID: \"09143401-93b0-453e-a738-9b8fa5457a81\") " pod="openshift-console-operator/console-operator-68b757fb67-62lvt" Jul 06 19:23:20.950938 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:20.950930 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbflh\" (UniqueName: \"kubernetes.io/projected/f9180d35-c91f-42b4-b49d-eebdf8b3a1c3-kube-api-access-wbflh\") pod \"volume-data-source-validator-5d456fdbcd-dqgwt\" (UID: \"f9180d35-c91f-42b4-b49d-eebdf8b3a1c3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dqgwt" Jul 06 19:23:21.039589 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:21.039551 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f02ed85e-784b-4f19-bd40-c79ebd77a8a9-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-rhbbf\" (UID: \"f02ed85e-784b-4f19-bd40-c79ebd77a8a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-rhbbf" Jul 06 19:23:21.039589 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:21.039588 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnfv5\" (UniqueName: \"kubernetes.io/projected/f02ed85e-784b-4f19-bd40-c79ebd77a8a9-kube-api-access-vnfv5\") pod \"kube-storage-version-migrator-operator-846bf668f7-rhbbf\" (UID: \"f02ed85e-784b-4f19-bd40-c79ebd77a8a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-rhbbf" Jul 06 19:23:21.039860 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:21.039636 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f02ed85e-784b-4f19-bd40-c79ebd77a8a9-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-rhbbf\" (UID: \"f02ed85e-784b-4f19-bd40-c79ebd77a8a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-rhbbf" Jul 06 19:23:21.040349 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:21.040311 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f02ed85e-784b-4f19-bd40-c79ebd77a8a9-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-rhbbf\" (UID: \"f02ed85e-784b-4f19-bd40-c79ebd77a8a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-rhbbf" Jul 06 19:23:21.042023 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:21.042000 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f02ed85e-784b-4f19-bd40-c79ebd77a8a9-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-rhbbf\" (UID: \"f02ed85e-784b-4f19-bd40-c79ebd77a8a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-rhbbf" Jul 06 19:23:21.047806 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:21.047784 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnfv5\" (UniqueName: \"kubernetes.io/projected/f02ed85e-784b-4f19-bd40-c79ebd77a8a9-kube-api-access-vnfv5\") pod \"kube-storage-version-migrator-operator-846bf668f7-rhbbf\" (UID: \"f02ed85e-784b-4f19-bd40-c79ebd77a8a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-rhbbf" Jul 06 19:23:21.090580 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:21.090543 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dqgwt" Jul 06 19:23:21.105706 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:21.105662 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-bx5pn" Jul 06 19:23:21.127498 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:21.127472 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-62lvt" Jul 06 19:23:21.194549 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:21.194517 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-rhbbf" Jul 06 19:23:21.444719 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:21.444614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c76c3d74-82a5-4c4c-9aab-876dd450bd83-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2gvph\" (UID: \"c76c3d74-82a5-4c4c-9aab-876dd450bd83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph" Jul 06 19:23:21.445146 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:21.444793 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:23:21.445146 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:21.444896 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c76c3d74-82a5-4c4c-9aab-876dd450bd83-samples-operator-tls podName:c76c3d74-82a5-4c4c-9aab-876dd450bd83 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:23.444872385 +0000 UTC m=+48.073826330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c76c3d74-82a5-4c4c-9aab-876dd450bd83-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2gvph" (UID: "c76c3d74-82a5-4c4c-9aab-876dd450bd83") : secret "samples-operator-tls" not found Jul 06 19:23:22.632905 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:22.632849 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-62lvt"] Jul 06 19:23:22.635808 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:22.635781 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09143401_93b0_453e_a738_9b8fa5457a81.slice/crio-ee07481207de485b62941472905b4b7df9390d57adef2d8c14b9675d24452ff6 WatchSource:0}: Error finding container ee07481207de485b62941472905b4b7df9390d57adef2d8c14b9675d24452ff6: Status 404 returned error can't find the container with id ee07481207de485b62941472905b4b7df9390d57adef2d8c14b9675d24452ff6 Jul 06 19:23:22.869366 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:22.869325 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dqgwt"] Jul 06 19:23:22.874462 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:22.874167 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-bx5pn"] Jul 06 19:23:22.875534 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:22.875505 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-rhbbf"] Jul 06 19:23:22.881861 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:22.881828 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02ed85e_784b_4f19_bd40_c79ebd77a8a9.slice/crio-7d1d0b816152e26dd7ab40fe94016e917bd6fb003e0f06236b622c357801a37b WatchSource:0}: Error finding container 7d1d0b816152e26dd7ab40fe94016e917bd6fb003e0f06236b622c357801a37b: Status 404 returned error can't find the container with id 7d1d0b816152e26dd7ab40fe94016e917bd6fb003e0f06236b622c357801a37b Jul 06 19:23:22.882434 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:22.882416 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9180d35_c91f_42b4_b49d_eebdf8b3a1c3.slice/crio-4ce9666c9b7d324c31a2d3e8c7c56bf1c9d9e5017d6caf1a2b01f15d7d6aa205 WatchSource:0}: Error finding container 4ce9666c9b7d324c31a2d3e8c7c56bf1c9d9e5017d6caf1a2b01f15d7d6aa205: Status 404 returned error can't find the container with id 4ce9666c9b7d324c31a2d3e8c7c56bf1c9d9e5017d6caf1a2b01f15d7d6aa205 Jul 06 19:23:22.884496 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:22.884476 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfccf3d5c_efcb_456c_8dc9_9aa7191184d4.slice/crio-c6482f6d8430ff942e76c9e15253a37c99abfa911872416472db67cd97b84bab WatchSource:0}: Error finding container c6482f6d8430ff942e76c9e15253a37c99abfa911872416472db67cd97b84bab: Status 404 returned error can't find the container with id c6482f6d8430ff942e76c9e15253a37c99abfa911872416472db67cd97b84bab Jul 06 19:23:23.104624 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.104580 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" event={"ID":"0f101d66-483d-43f5-89f7-cb8dcc39f131","Type":"ContainerStarted","Data":"a8350c8e77493dee37f29bd43a501c4140051bc4a639105edae5bf979221befb"} Jul 06 19:23:23.104815 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.104639 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" event={"ID":"0f101d66-483d-43f5-89f7-cb8dcc39f131","Type":"ContainerStarted","Data":"4a85caddbdfb228fb2f3b340c5e9d8e1851af3f0f00783d83f8296e676eda165"} Jul 06 19:23:23.104815 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.104654 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" event={"ID":"0f101d66-483d-43f5-89f7-cb8dcc39f131","Type":"ContainerStarted","Data":"bbebe50c3c6154ecd826747db50e47c062af20d37be3014c30b745ce1833a96b"} Jul 06 19:23:23.106091 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.106064 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" event={"ID":"7bf04227-6bc5-4797-a189-f1c12841bfc4","Type":"ContainerStarted","Data":"b8945a00701998363c73c1f26073c3464e6ee03e72e7b4a6bfaf1cca94d36605"} Jul 06 19:23:23.107187 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.107149 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-bx5pn" event={"ID":"fccf3d5c-efcb-456c-8dc9-9aa7191184d4","Type":"ContainerStarted","Data":"c6482f6d8430ff942e76c9e15253a37c99abfa911872416472db67cd97b84bab"} Jul 06 19:23:23.108520 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.108495 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k" event={"ID":"550ace7a-6e14-4cd8-9470-54099bdac0e3","Type":"ContainerStarted","Data":"e079e61987603077d0d786ec38e6c19e24bca39b53d26aa1a5e14aba594d63eb"} Jul 06 19:23:23.109948 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.109921 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dk4kn" event={"ID":"07c083da-b946-4438-afa0-3c5be0f77b9b","Type":"ContainerStarted","Data":"77ccb753c512c4204b3704cf82aa752b0f28355fbcbaf6733da8d5a359950e1d"} Jul 06 19:23:23.110934 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.110918 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-62lvt" event={"ID":"09143401-93b0-453e-a738-9b8fa5457a81","Type":"ContainerStarted","Data":"ee07481207de485b62941472905b4b7df9390d57adef2d8c14b9675d24452ff6"} Jul 06 19:23:23.112021 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.112002 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-rhbbf" event={"ID":"f02ed85e-784b-4f19-bd40-c79ebd77a8a9","Type":"ContainerStarted","Data":"7d1d0b816152e26dd7ab40fe94016e917bd6fb003e0f06236b622c357801a37b"} Jul 06 19:23:23.113327 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.113303 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c75b6496c-gpck7" event={"ID":"586de6b0-7270-4ddf-b643-d89d0d21e61e","Type":"ContainerStarted","Data":"f45a8dc0a12cd005146fc331ef0e07dfbf7c574fdc1fa0cb0885ea24f1d81171"} Jul 06 19:23:23.113477 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.113463 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c75b6496c-gpck7" Jul 06 19:23:23.114472 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.114451 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dqgwt" event={"ID":"f9180d35-c91f-42b4-b49d-eebdf8b3a1c3","Type":"ContainerStarted","Data":"4ce9666c9b7d324c31a2d3e8c7c56bf1c9d9e5017d6caf1a2b01f15d7d6aa205"} Jul 06 19:23:23.116173 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.116151 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c75b6496c-gpck7" Jul 06 19:23:23.130475 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.130430 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" podStartSLOduration=2.355502494 podStartE2EDuration="11.130415402s" totalImagesPullingTime="8.774912908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:23.129719571 +0000 UTC m=+47.758673538" watchObservedRunningTime="2026-07-06 19:23:23.130415402 +0000 UTC m=+47.759369345" Jul 06 19:23:23.158744 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.158669 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c75b6496c-gpck7" podStartSLOduration=2.392822897 podStartE2EDuration="11.15864665s" totalImagesPullingTime="8.765823753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:23.158184375 +0000 UTC m=+47.787138341" watchObservedRunningTime="2026-07-06 19:23:23.15864665 +0000 UTC m=+47.787600618" Jul 06 19:23:23.207595 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.207448 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" podStartSLOduration=4.127248091 podStartE2EDuration="13.207424659s" totalImagesPullingTime="9.080176568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:23.184834872 +0000 UTC m=+47.813788838" watchObservedRunningTime="2026-07-06 19:23:23.207424659 +0000 UTC m=+47.836378626" Jul 06 19:23:23.207981 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.207945 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dk4kn" podStartSLOduration=3.119467825 podStartE2EDuration="12.207934348s" totalImagesPullingTime="9.088466523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:23.207311432 +0000 UTC m=+47.836265400" watchObservedRunningTime="2026-07-06 19:23:23.207934348 +0000 UTC m=+47.836888314" Jul 06 19:23:23.245729 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.245608 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd9476ff6-4rt8k" podStartSLOduration=2.480196156 podStartE2EDuration="11.245588943s" totalImagesPullingTime="8.765392787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:23.244123867 +0000 UTC m=+47.873077834" watchObservedRunningTime="2026-07-06 19:23:23.245588943 +0000 UTC m=+47.874542908" Jul 06 19:23:23.467112 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.467078 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c76c3d74-82a5-4c4c-9aab-876dd450bd83-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2gvph\" (UID: \"c76c3d74-82a5-4c4c-9aab-876dd450bd83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph" Jul 06 19:23:23.467250 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:23.467216 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:23:23.467316 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:23.467300 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c76c3d74-82a5-4c4c-9aab-876dd450bd83-samples-operator-tls podName:c76c3d74-82a5-4c4c-9aab-876dd450bd83 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:27.467278383 +0000 UTC m=+52.096232343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c76c3d74-82a5-4c4c-9aab-876dd450bd83-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2gvph" (UID: "c76c3d74-82a5-4c4c-9aab-876dd450bd83") : secret "samples-operator-tls" not found Jul 06 19:23:23.935441 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.933856 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cxsxv"] Jul 06 19:23:23.953605 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.953570 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:23.964385 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.960583 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 19:23:23.966466 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.965337 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lq9xg\"" Jul 06 19:23:23.966466 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.965805 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 19:23:23.967165 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:23.967007 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cxsxv"] Jul 06 19:23:24.073089 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:24.073046 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtvx4\" (UniqueName: \"kubernetes.io/projected/61241f90-6bd8-4a8c-8784-02757d7fcb36-kube-api-access-qtvx4\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:24.073374 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:24.073168 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61241f90-6bd8-4a8c-8784-02757d7fcb36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:24.073492 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:24.073385 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61241f90-6bd8-4a8c-8784-02757d7fcb36-data-volume\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:24.073492 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:24.073460 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61241f90-6bd8-4a8c-8784-02757d7fcb36-crio-socket\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:24.073608 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:24.073578 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61241f90-6bd8-4a8c-8784-02757d7fcb36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:24.174677 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:24.174638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtvx4\" (UniqueName: \"kubernetes.io/projected/61241f90-6bd8-4a8c-8784-02757d7fcb36-kube-api-access-qtvx4\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:24.174873 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:24.174802 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61241f90-6bd8-4a8c-8784-02757d7fcb36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:24.175145 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:24.175119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61241f90-6bd8-4a8c-8784-02757d7fcb36-data-volume\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:24.175216 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:24.175188 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61241f90-6bd8-4a8c-8784-02757d7fcb36-crio-socket\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:24.175331 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:24.175313 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61241f90-6bd8-4a8c-8784-02757d7fcb36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:24.175718 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:24.175684 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61241f90-6bd8-4a8c-8784-02757d7fcb36-data-volume\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:24.177221 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:24.175801 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:23:24.177221 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:24.175876 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61241f90-6bd8-4a8c-8784-02757d7fcb36-kube-rbac-proxy-cm podName:61241f90-6bd8-4a8c-8784-02757d7fcb36 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:24.675854983 +0000 UTC m=+49.304808941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/61241f90-6bd8-4a8c-8784-02757d7fcb36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cxsxv" (UID: "61241f90-6bd8-4a8c-8784-02757d7fcb36") : configmap "kube-rbac-proxy" not found Jul 06 19:23:24.177221 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:24.176302 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61241f90-6bd8-4a8c-8784-02757d7fcb36-crio-socket\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:24.177221 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:24.176862 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:23:24.177221 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:24.176908 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61241f90-6bd8-4a8c-8784-02757d7fcb36-insights-runtime-extractor-tls podName:61241f90-6bd8-4a8c-8784-02757d7fcb36 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:24.676892748 +0000 UTC m=+49.305846695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61241f90-6bd8-4a8c-8784-02757d7fcb36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cxsxv" (UID: "61241f90-6bd8-4a8c-8784-02757d7fcb36") : secret "insights-runtime-extractor-tls" not found Jul 06 19:23:24.197509 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:24.197401 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtvx4\" (UniqueName: \"kubernetes.io/projected/61241f90-6bd8-4a8c-8784-02757d7fcb36-kube-api-access-qtvx4\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:24.680155 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:24.680113 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61241f90-6bd8-4a8c-8784-02757d7fcb36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:24.680353 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:24.680230 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:23:24.680353 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:24.680280 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:23:24.680353 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:24.680234 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61241f90-6bd8-4a8c-8784-02757d7fcb36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:24.680353 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:24.680312 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61241f90-6bd8-4a8c-8784-02757d7fcb36-insights-runtime-extractor-tls podName:61241f90-6bd8-4a8c-8784-02757d7fcb36 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:25.680289244 +0000 UTC m=+50.309243193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61241f90-6bd8-4a8c-8784-02757d7fcb36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cxsxv" (UID: "61241f90-6bd8-4a8c-8784-02757d7fcb36") : secret "insights-runtime-extractor-tls" not found Jul 06 19:23:24.680353 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:24.680340 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61241f90-6bd8-4a8c-8784-02757d7fcb36-kube-rbac-proxy-cm podName:61241f90-6bd8-4a8c-8784-02757d7fcb36 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:25.680329908 +0000 UTC m=+50.309283855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/61241f90-6bd8-4a8c-8784-02757d7fcb36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cxsxv" (UID: "61241f90-6bd8-4a8c-8784-02757d7fcb36") : configmap "kube-rbac-proxy" not found Jul 06 19:23:25.329816 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:25.329785 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tnbvj_8c3ea532-8f74-47ba-bd67-89fc5cd148c3/dns-node-resolver/0.log" Jul 06 19:23:25.690333 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:25.690242 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61241f90-6bd8-4a8c-8784-02757d7fcb36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:25.690496 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:25.690340 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61241f90-6bd8-4a8c-8784-02757d7fcb36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:25.690496 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:25.690344 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:23:25.690496 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:25.690416 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61241f90-6bd8-4a8c-8784-02757d7fcb36-insights-runtime-extractor-tls podName:61241f90-6bd8-4a8c-8784-02757d7fcb36 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:27.69040006 +0000 UTC m=+52.319354003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61241f90-6bd8-4a8c-8784-02757d7fcb36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cxsxv" (UID: "61241f90-6bd8-4a8c-8784-02757d7fcb36") : secret "insights-runtime-extractor-tls" not found Jul 06 19:23:25.690496 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:25.690453 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:23:25.690723 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:25.690502 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61241f90-6bd8-4a8c-8784-02757d7fcb36-kube-rbac-proxy-cm podName:61241f90-6bd8-4a8c-8784-02757d7fcb36 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:27.69048996 +0000 UTC m=+52.319443904 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/61241f90-6bd8-4a8c-8784-02757d7fcb36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cxsxv" (UID: "61241f90-6bd8-4a8c-8784-02757d7fcb36") : configmap "kube-rbac-proxy" not found Jul 06 19:23:26.142541 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:26.142516 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qfm4f_ecbc6515-e4c1-489d-ac42-dd3a967b782c/node-ca/0.log" Jul 06 19:23:26.661148 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:26.661128 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-mf7qg"] Jul 06 19:23:26.676614 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:26.676588 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-mf7qg"] Jul 06 19:23:26.676764 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:26.676717 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-mf7qg" Jul 06 19:23:26.678870 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:26.678850 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-96j2r\"" Jul 06 19:23:26.729824 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:26.729803 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dk4kn_07c083da-b946-4438-afa0-3c5be0f77b9b/serve-healthcheck-canary/0.log" Jul 06 19:23:26.801492 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:26.801458 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rswsc\" (UniqueName: \"kubernetes.io/projected/36b3d96c-e06a-4a64-b7df-865f351ed59f-kube-api-access-rswsc\") pod \"network-check-source-989f8b8dc-mf7qg\" (UID: \"36b3d96c-e06a-4a64-b7df-865f351ed59f\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-mf7qg" Jul 06 19:23:26.902123 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:26.902037 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rswsc\" (UniqueName: \"kubernetes.io/projected/36b3d96c-e06a-4a64-b7df-865f351ed59f-kube-api-access-rswsc\") pod \"network-check-source-989f8b8dc-mf7qg\" (UID: \"36b3d96c-e06a-4a64-b7df-865f351ed59f\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-mf7qg" Jul 06 19:23:26.911068 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:26.911036 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rswsc\" (UniqueName: \"kubernetes.io/projected/36b3d96c-e06a-4a64-b7df-865f351ed59f-kube-api-access-rswsc\") pod \"network-check-source-989f8b8dc-mf7qg\" (UID: \"36b3d96c-e06a-4a64-b7df-865f351ed59f\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-mf7qg" Jul 06 19:23:27.025404 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.025374 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-mf7qg" Jul 06 19:23:27.133785 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.133312 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-bx5pn" event={"ID":"fccf3d5c-efcb-456c-8dc9-9aa7191184d4","Type":"ContainerStarted","Data":"59156710fdcea168d54b0102301834972a6c4d895b7be1837ab3dedcbe677c46"} Jul 06 19:23:27.135664 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.135324 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/0.log" Jul 06 19:23:27.135664 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.135363 2578 generic.go:356] "Generic (PLEG): container finished" podID="09143401-93b0-453e-a738-9b8fa5457a81" containerID="04b202c08a96604f3c1ec5abfe5090d14fe882521f3c08f848e285ce2aeac6b4" exitCode=255 Jul 06 19:23:27.135664 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.135433 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-62lvt" event={"ID":"09143401-93b0-453e-a738-9b8fa5457a81","Type":"ContainerDied","Data":"04b202c08a96604f3c1ec5abfe5090d14fe882521f3c08f848e285ce2aeac6b4"} Jul 06 19:23:27.135932 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.135783 2578 scope.go:117] "RemoveContainer" containerID="04b202c08a96604f3c1ec5abfe5090d14fe882521f3c08f848e285ce2aeac6b4" Jul 06 19:23:27.139225 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.138757 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-rhbbf" event={"ID":"f02ed85e-784b-4f19-bd40-c79ebd77a8a9","Type":"ContainerStarted","Data":"fe5cef041480ec2372362f8c10a0b6a266180b3fcc072378aa0a22ebe531626d"} Jul 06 19:23:27.141068 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.140781 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dqgwt" event={"ID":"f9180d35-c91f-42b4-b49d-eebdf8b3a1c3","Type":"ContainerStarted","Data":"65412b661943adc717df72a7f363a58bd051482d0c83eaf7799500c3efd61c6d"} Jul 06 19:23:27.161393 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.161337 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-bx5pn" podStartSLOduration=3.40350403 podStartE2EDuration="7.161316067s" totalImagesPullingTime="3.757812037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:27.16015099 +0000 UTC m=+51.789104959" watchObservedRunningTime="2026-07-06 19:23:27.161316067 +0000 UTC m=+51.790270035" Jul 06 19:23:27.195086 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.194979 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-mf7qg"] Jul 06 19:23:27.197422 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:27.197382 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b3d96c_e06a_4a64_b7df_865f351ed59f.slice/crio-8944c7600d28b1d464e846174bf68f6b7f5579158af2777ca24b3119504b536b WatchSource:0}: Error finding container 8944c7600d28b1d464e846174bf68f6b7f5579158af2777ca24b3119504b536b: Status 404 returned error can't find the container with id 8944c7600d28b1d464e846174bf68f6b7f5579158af2777ca24b3119504b536b Jul 06 19:23:27.203053 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.203009 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-rhbbf" podStartSLOduration=3.108152865 podStartE2EDuration="7.202993444s" totalImagesPullingTime="4.094840579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:27.201895489 +0000 UTC m=+51.830849456" watchObservedRunningTime="2026-07-06 19:23:27.202993444 +0000 UTC m=+51.831947416" Jul 06 19:23:27.205322 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.205302 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:27.205477 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:27.205459 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:23:27.205555 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:27.205481 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-954dc4658-xvp8b: secret "image-registry-tls" not found Jul 06 19:23:27.205555 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:27.205534 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls podName:ee43e118-575d-4aa1-9276-f9f1ed3606a4 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:43.205513977 +0000 UTC m=+67.834467940 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls") pod "image-registry-954dc4658-xvp8b" (UID: "ee43e118-575d-4aa1-9276-f9f1ed3606a4") : secret "image-registry-tls" not found Jul 06 19:23:27.221758 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.221684 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dqgwt" podStartSLOduration=3.47064132 podStartE2EDuration="7.221660733s" totalImagesPullingTime="3.751019413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:27.21967424 +0000 UTC m=+51.848628206" watchObservedRunningTime="2026-07-06 19:23:27.221660733 +0000 UTC m=+51.850614704" Jul 06 19:23:27.306830 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.306798 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:27.306978 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.306895 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:27.307054 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.306974 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-z8zqc\" (UID: \"8aa685e5-fc07-4d7b-bb90-506c9c9f34be\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" Jul 06 19:23:27.307109 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:27.307055 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle podName:cc56f897-5d59-4451-a517-5bfea063ce0c nodeName:}" failed. No retries permitted until 2026-07-06 19:23:43.307035678 +0000 UTC m=+67.935989622 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle") pod "router-default-68d9d48ccc-kk7pz" (UID: "cc56f897-5d59-4451-a517-5bfea063ce0c") : configmap references non-existent config key: service-ca.crt Jul 06 19:23:27.307109 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:27.307069 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 19:23:27.307201 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:27.307140 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs podName:cc56f897-5d59-4451-a517-5bfea063ce0c nodeName:}" failed. No retries permitted until 2026-07-06 19:23:43.307122029 +0000 UTC m=+67.936075989 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs") pod "router-default-68d9d48ccc-kk7pz" (UID: "cc56f897-5d59-4451-a517-5bfea063ce0c") : secret "router-metrics-certs-default" not found Jul 06 19:23:27.307201 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:27.307074 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 19:23:27.307267 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:27.307232 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls podName:8aa685e5-fc07-4d7b-bb90-506c9c9f34be nodeName:}" failed. No retries permitted until 2026-07-06 19:23:43.307214711 +0000 UTC m=+67.936168660 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-z8zqc" (UID: "8aa685e5-fc07-4d7b-bb90-506c9c9f34be") : secret "cluster-monitoring-operator-tls" not found Jul 06 19:23:27.408212 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.408117 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls\") pod \"dns-default-8j849\" (UID: \"670f9fdc-7f0b-44f5-9277-c88477bde835\") " pod="openshift-dns/dns-default-8j849" Jul 06 19:23:27.408361 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:27.408305 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:23:27.408406 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:27.408388 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls podName:670f9fdc-7f0b-44f5-9277-c88477bde835 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:43.408366772 +0000 UTC m=+68.037320723 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls") pod "dns-default-8j849" (UID: "670f9fdc-7f0b-44f5-9277-c88477bde835") : secret "dns-default-metrics-tls" not found Jul 06 19:23:27.509669 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.509637 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c76c3d74-82a5-4c4c-9aab-876dd450bd83-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2gvph\" (UID: \"c76c3d74-82a5-4c4c-9aab-876dd450bd83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph" Jul 06 19:23:27.509910 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:27.509810 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:23:27.509910 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:27.509876 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c76c3d74-82a5-4c4c-9aab-876dd450bd83-samples-operator-tls podName:c76c3d74-82a5-4c4c-9aab-876dd450bd83 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:35.509858198 +0000 UTC m=+60.138812158 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c76c3d74-82a5-4c4c-9aab-876dd450bd83-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2gvph" (UID: "c76c3d74-82a5-4c4c-9aab-876dd450bd83") : secret "samples-operator-tls" not found Jul 06 19:23:27.712366 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.712266 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61241f90-6bd8-4a8c-8784-02757d7fcb36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:27.712786 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.712376 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61241f90-6bd8-4a8c-8784-02757d7fcb36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:27.712786 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:27.712428 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:23:27.712786 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:27.712507 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61241f90-6bd8-4a8c-8784-02757d7fcb36-insights-runtime-extractor-tls podName:61241f90-6bd8-4a8c-8784-02757d7fcb36 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:31.712482918 +0000 UTC m=+56.341436886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61241f90-6bd8-4a8c-8784-02757d7fcb36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cxsxv" (UID: "61241f90-6bd8-4a8c-8784-02757d7fcb36") : secret "insights-runtime-extractor-tls" not found Jul 06 19:23:27.712944 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:27.712927 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61241f90-6bd8-4a8c-8784-02757d7fcb36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:28.145873 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:28.145799 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-mf7qg" event={"ID":"36b3d96c-e06a-4a64-b7df-865f351ed59f","Type":"ContainerStarted","Data":"8944c7600d28b1d464e846174bf68f6b7f5579158af2777ca24b3119504b536b"} Jul 06 19:23:28.147669 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:28.147641 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 19:23:28.148303 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:28.148283 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/0.log" Jul 06 19:23:28.148391 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:28.148324 2578 generic.go:356] "Generic (PLEG): container finished" podID="09143401-93b0-453e-a738-9b8fa5457a81" containerID="502afffb18f0176ddb764a418fe4d2b67067ce854f00675317a02f1b8bae697b" exitCode=255 Jul 06 19:23:28.148447 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:28.148419 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-62lvt" event={"ID":"09143401-93b0-453e-a738-9b8fa5457a81","Type":"ContainerDied","Data":"502afffb18f0176ddb764a418fe4d2b67067ce854f00675317a02f1b8bae697b"} Jul 06 19:23:28.148508 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:28.148470 2578 scope.go:117] "RemoveContainer" containerID="04b202c08a96604f3c1ec5abfe5090d14fe882521f3c08f848e285ce2aeac6b4" Jul 06 19:23:28.149165 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:28.148740 2578 scope.go:117] "RemoveContainer" containerID="502afffb18f0176ddb764a418fe4d2b67067ce854f00675317a02f1b8bae697b" Jul 06 19:23:28.149165 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:28.148930 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-62lvt_openshift-console-operator(09143401-93b0-453e-a738-9b8fa5457a81)\"" pod="openshift-console-operator/console-operator-68b757fb67-62lvt" podUID="09143401-93b0-453e-a738-9b8fa5457a81" Jul 06 19:23:29.153822 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:29.153789 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 19:23:29.154351 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:29.154203 2578 scope.go:117] "RemoveContainer" containerID="502afffb18f0176ddb764a418fe4d2b67067ce854f00675317a02f1b8bae697b" Jul 06 19:23:29.154430 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:29.154396 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-62lvt_openshift-console-operator(09143401-93b0-453e-a738-9b8fa5457a81)\"" pod="openshift-console-operator/console-operator-68b757fb67-62lvt" podUID="09143401-93b0-453e-a738-9b8fa5457a81" Jul 06 19:23:29.647923 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:29.647878 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-4w4p4"] Jul 06 19:23:29.671577 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:29.671547 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-4w4p4"] Jul 06 19:23:29.671775 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:29.671716 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4w4p4" Jul 06 19:23:29.674268 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:29.674104 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 06 19:23:29.674268 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:29.674168 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 06 19:23:29.674268 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:29.674257 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zdzsh\"" Jul 06 19:23:29.734706 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:29.734653 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hbjw\" (UniqueName: \"kubernetes.io/projected/231caf7f-18fb-4b59-95a9-7e19521d406b-kube-api-access-2hbjw\") pod \"migrator-d57d58cc9-4w4p4\" (UID: \"231caf7f-18fb-4b59-95a9-7e19521d406b\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4w4p4" Jul 06 19:23:29.835992 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:29.835952 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hbjw\" (UniqueName: \"kubernetes.io/projected/231caf7f-18fb-4b59-95a9-7e19521d406b-kube-api-access-2hbjw\") pod \"migrator-d57d58cc9-4w4p4\" (UID: \"231caf7f-18fb-4b59-95a9-7e19521d406b\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4w4p4" Jul 06 19:23:29.856635 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:29.856608 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hbjw\" (UniqueName: \"kubernetes.io/projected/231caf7f-18fb-4b59-95a9-7e19521d406b-kube-api-access-2hbjw\") pod \"migrator-d57d58cc9-4w4p4\" (UID: \"231caf7f-18fb-4b59-95a9-7e19521d406b\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4w4p4" Jul 06 19:23:29.982531 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:29.982433 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4w4p4" Jul 06 19:23:30.472963 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:30.472922 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231caf7f_18fb_4b59_95a9_7e19521d406b.slice/crio-b6714bcd08ad7ee30dc3ca3e9f0c0fa14f04bcc6d6855a6f171fc0b20173db62 WatchSource:0}: Error finding container b6714bcd08ad7ee30dc3ca3e9f0c0fa14f04bcc6d6855a6f171fc0b20173db62: Status 404 returned error can't find the container with id b6714bcd08ad7ee30dc3ca3e9f0c0fa14f04bcc6d6855a6f171fc0b20173db62 Jul 06 19:23:30.476209 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:30.476189 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-4w4p4"] Jul 06 19:23:31.080020 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.079977 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-tm7lp"] Jul 06 19:23:31.082243 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.082225 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-tm7lp" Jul 06 19:23:31.085103 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.085083 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 06 19:23:31.085776 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.085762 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 06 19:23:31.086006 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.085991 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 06 19:23:31.086059 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.086027 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 06 19:23:31.086755 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.086740 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zgnh9\"" Jul 06 19:23:31.104063 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.104028 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-tm7lp"] Jul 06 19:23:31.128174 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.128142 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-62lvt" Jul 06 19:23:31.128371 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.128202 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-62lvt" Jul 06 19:23:31.128641 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.128617 2578 scope.go:117] "RemoveContainer" containerID="502afffb18f0176ddb764a418fe4d2b67067ce854f00675317a02f1b8bae697b" Jul 06 19:23:31.128868 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:31.128848 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-62lvt_openshift-console-operator(09143401-93b0-453e-a738-9b8fa5457a81)\"" pod="openshift-console-operator/console-operator-68b757fb67-62lvt" podUID="09143401-93b0-453e-a738-9b8fa5457a81" Jul 06 19:23:31.150033 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.149998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/29d49792-6bb5-4950-9a54-bf47433c8b5e-signing-cabundle\") pod \"service-ca-758f599c5-tm7lp\" (UID: \"29d49792-6bb5-4950-9a54-bf47433c8b5e\") " pod="openshift-service-ca/service-ca-758f599c5-tm7lp" Jul 06 19:23:31.150236 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.150051 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/29d49792-6bb5-4950-9a54-bf47433c8b5e-signing-key\") pod \"service-ca-758f599c5-tm7lp\" (UID: \"29d49792-6bb5-4950-9a54-bf47433c8b5e\") " pod="openshift-service-ca/service-ca-758f599c5-tm7lp" Jul 06 19:23:31.150236 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.150123 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk82h\" (UniqueName: \"kubernetes.io/projected/29d49792-6bb5-4950-9a54-bf47433c8b5e-kube-api-access-mk82h\") pod \"service-ca-758f599c5-tm7lp\" (UID: \"29d49792-6bb5-4950-9a54-bf47433c8b5e\") " pod="openshift-service-ca/service-ca-758f599c5-tm7lp" Jul 06 19:23:31.162076 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.162036 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4w4p4" event={"ID":"231caf7f-18fb-4b59-95a9-7e19521d406b","Type":"ContainerStarted","Data":"b6714bcd08ad7ee30dc3ca3e9f0c0fa14f04bcc6d6855a6f171fc0b20173db62"} Jul 06 19:23:31.163864 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.163822 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-mf7qg" event={"ID":"36b3d96c-e06a-4a64-b7df-865f351ed59f","Type":"ContainerStarted","Data":"704510f40403c80d2b52d5bd76d4f1538541fb01e3ebcbad7d0ebbb83b34407a"} Jul 06 19:23:31.182521 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.182454 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-mf7qg" podStartSLOduration=2.017650364 podStartE2EDuration="5.182430359s" totalImagesPullingTime="3.164779995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:31.180684697 +0000 UTC m=+55.809638665" watchObservedRunningTime="2026-07-06 19:23:31.182430359 +0000 UTC m=+55.811384328" Jul 06 19:23:31.251472 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.251437 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/29d49792-6bb5-4950-9a54-bf47433c8b5e-signing-cabundle\") pod \"service-ca-758f599c5-tm7lp\" (UID: \"29d49792-6bb5-4950-9a54-bf47433c8b5e\") " pod="openshift-service-ca/service-ca-758f599c5-tm7lp" Jul 06 19:23:31.251472 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.251481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/29d49792-6bb5-4950-9a54-bf47433c8b5e-signing-key\") pod \"service-ca-758f599c5-tm7lp\" (UID: \"29d49792-6bb5-4950-9a54-bf47433c8b5e\") " pod="openshift-service-ca/service-ca-758f599c5-tm7lp" Jul 06 19:23:31.251687 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.251514 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk82h\" (UniqueName: \"kubernetes.io/projected/29d49792-6bb5-4950-9a54-bf47433c8b5e-kube-api-access-mk82h\") pod \"service-ca-758f599c5-tm7lp\" (UID: \"29d49792-6bb5-4950-9a54-bf47433c8b5e\") " pod="openshift-service-ca/service-ca-758f599c5-tm7lp" Jul 06 19:23:31.252285 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.252256 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/29d49792-6bb5-4950-9a54-bf47433c8b5e-signing-cabundle\") pod \"service-ca-758f599c5-tm7lp\" (UID: \"29d49792-6bb5-4950-9a54-bf47433c8b5e\") " pod="openshift-service-ca/service-ca-758f599c5-tm7lp" Jul 06 19:23:31.254131 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.254111 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/29d49792-6bb5-4950-9a54-bf47433c8b5e-signing-key\") pod \"service-ca-758f599c5-tm7lp\" (UID: \"29d49792-6bb5-4950-9a54-bf47433c8b5e\") " pod="openshift-service-ca/service-ca-758f599c5-tm7lp" Jul 06 19:23:31.260336 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.260311 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk82h\" (UniqueName: \"kubernetes.io/projected/29d49792-6bb5-4950-9a54-bf47433c8b5e-kube-api-access-mk82h\") pod \"service-ca-758f599c5-tm7lp\" (UID: \"29d49792-6bb5-4950-9a54-bf47433c8b5e\") " pod="openshift-service-ca/service-ca-758f599c5-tm7lp" Jul 06 19:23:31.391523 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.391429 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-tm7lp" Jul 06 19:23:31.757204 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.757156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61241f90-6bd8-4a8c-8784-02757d7fcb36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:31.757530 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:31.757290 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:23:31.757530 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:23:31.757363 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61241f90-6bd8-4a8c-8784-02757d7fcb36-insights-runtime-extractor-tls podName:61241f90-6bd8-4a8c-8784-02757d7fcb36 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:39.757343639 +0000 UTC m=+64.386297599 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61241f90-6bd8-4a8c-8784-02757d7fcb36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cxsxv" (UID: "61241f90-6bd8-4a8c-8784-02757d7fcb36") : secret "insights-runtime-extractor-tls" not found Jul 06 19:23:31.759540 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:31.759520 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-tm7lp"] Jul 06 19:23:31.762230 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:31.762176 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d49792_6bb5_4950_9a54_bf47433c8b5e.slice/crio-db48fb4a5a63850bbde1556fcdd6463dfbc12c2877f93b778497f5f2c8f3d005 WatchSource:0}: Error finding container db48fb4a5a63850bbde1556fcdd6463dfbc12c2877f93b778497f5f2c8f3d005: Status 404 returned error can't find the container with id db48fb4a5a63850bbde1556fcdd6463dfbc12c2877f93b778497f5f2c8f3d005 Jul 06 19:23:32.168862 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:32.168820 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-tm7lp" event={"ID":"29d49792-6bb5-4950-9a54-bf47433c8b5e","Type":"ContainerStarted","Data":"c56b0bca14b3810ca9d58e2ae44352f17edbbcddcb8772f745d12ea1938f73eb"} Jul 06 19:23:32.169065 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:32.168884 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-tm7lp" event={"ID":"29d49792-6bb5-4950-9a54-bf47433c8b5e","Type":"ContainerStarted","Data":"db48fb4a5a63850bbde1556fcdd6463dfbc12c2877f93b778497f5f2c8f3d005"} Jul 06 19:23:32.170687 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:32.170655 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4w4p4" event={"ID":"231caf7f-18fb-4b59-95a9-7e19521d406b","Type":"ContainerStarted","Data":"b6cbb0f29e9fc35766f05135ea3a6b081ab766151b7931c8c024e2c2ba690cd5"} Jul 06 19:23:32.170826 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:32.170750 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4w4p4" event={"ID":"231caf7f-18fb-4b59-95a9-7e19521d406b","Type":"ContainerStarted","Data":"4ae9d6c48c25a717389d46dd5b6dcd37b31ae08258d777002a3317448fc45ba6"} Jul 06 19:23:32.185022 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:32.184969 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-tm7lp" podStartSLOduration=1.18495294 podStartE2EDuration="1.18495294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:32.184093989 +0000 UTC m=+56.813047953" watchObservedRunningTime="2026-07-06 19:23:32.18495294 +0000 UTC m=+56.813906919" Jul 06 19:23:32.202143 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:32.202088 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4w4p4" podStartSLOduration=2.046347152 podStartE2EDuration="3.202071304s" totalImagesPullingTime="1.155724152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:32.200746091 +0000 UTC m=+56.829700058" watchObservedRunningTime="2026-07-06 19:23:32.202071304 +0000 UTC m=+56.831025269" Jul 06 19:23:35.596374 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:35.596335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c76c3d74-82a5-4c4c-9aab-876dd450bd83-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2gvph\" (UID: \"c76c3d74-82a5-4c4c-9aab-876dd450bd83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph" Jul 06 19:23:35.598796 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:35.598775 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c76c3d74-82a5-4c4c-9aab-876dd450bd83-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2gvph\" (UID: \"c76c3d74-82a5-4c4c-9aab-876dd450bd83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph" Jul 06 19:23:35.645229 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:35.645192 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph" Jul 06 19:23:35.845096 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:35.845062 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph"] Jul 06 19:23:36.028372 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:36.028342 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqwfz" Jul 06 19:23:36.185586 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:36.185502 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph" event={"ID":"c76c3d74-82a5-4c4c-9aab-876dd450bd83","Type":"ContainerStarted","Data":"c529659a05c92c94c89467c4987f71397ef3b7287c63bf7368b9590c098bf6d9"} Jul 06 19:23:38.194606 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:38.194509 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph" event={"ID":"c76c3d74-82a5-4c4c-9aab-876dd450bd83","Type":"ContainerStarted","Data":"333716d20d60bec18ffc651d19cca23d3b474fc2abb90ba9532c06161e95b172"} Jul 06 19:23:38.195075 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:38.194596 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph" event={"ID":"c76c3d74-82a5-4c4c-9aab-876dd450bd83","Type":"ContainerStarted","Data":"22b0fcbaba4036631a6d1dcf7b7a7f72ae807cdbae6e20d11d896099df352771"} Jul 06 19:23:38.237228 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:38.237175 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2gvph" podStartSLOduration=17.29582745 podStartE2EDuration="19.237157829s" totalImagesPullingTime="1.941330379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:38.236092136 +0000 UTC m=+62.865046103" watchObservedRunningTime="2026-07-06 19:23:38.237157829 +0000 UTC m=+62.866111845" Jul 06 19:23:39.839408 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:39.839358 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61241f90-6bd8-4a8c-8784-02757d7fcb36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:39.841738 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:39.841715 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61241f90-6bd8-4a8c-8784-02757d7fcb36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cxsxv\" (UID: \"61241f90-6bd8-4a8c-8784-02757d7fcb36\") " pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:39.872771 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:39.872742 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lq9xg\"" Jul 06 19:23:39.880613 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:39.880590 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cxsxv" Jul 06 19:23:40.020630 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:40.020606 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cxsxv"] Jul 06 19:23:40.022956 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:40.022921 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61241f90_6bd8_4a8c_8784_02757d7fcb36.slice/crio-25a16a5ef6e69c30f3c5174b6955b235040d60df934d8ae0c2465ef888bd4a6f WatchSource:0}: Error finding container 25a16a5ef6e69c30f3c5174b6955b235040d60df934d8ae0c2465ef888bd4a6f: Status 404 returned error can't find the container with id 25a16a5ef6e69c30f3c5174b6955b235040d60df934d8ae0c2465ef888bd4a6f Jul 06 19:23:40.202541 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:40.202459 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cxsxv" event={"ID":"61241f90-6bd8-4a8c-8784-02757d7fcb36","Type":"ContainerStarted","Data":"25a16a5ef6e69c30f3c5174b6955b235040d60df934d8ae0c2465ef888bd4a6f"} Jul 06 19:23:41.559897 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:41.559840 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs\") pod \"network-metrics-daemon-lh7hd\" (UID: \"362f1775-4ab7-400a-b216-a22e4ca6c9fb\") " pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:23:41.562855 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:41.562827 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:23:41.573569 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:41.573531 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/362f1775-4ab7-400a-b216-a22e4ca6c9fb-metrics-certs\") pod \"network-metrics-daemon-lh7hd\" (UID: \"362f1775-4ab7-400a-b216-a22e4ca6c9fb\") " pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:23:41.661492 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:41.661446 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret\") pod \"global-pull-secret-syncer-kpsb4\" (UID: \"a6d9fe54-0654-4f9d-bb79-0fbeec14c98f\") " pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:23:41.661663 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:41.661538 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pp42\" (UniqueName: \"kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42\") pod \"network-check-target-vfp2h\" (UID: \"abb3d52d-bb53-47f0-9708-075b2eae0e2d\") " pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:23:41.664464 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:41.664433 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pp42\" (UniqueName: \"kubernetes.io/projected/abb3d52d-bb53-47f0-9708-075b2eae0e2d-kube-api-access-8pp42\") pod \"network-check-target-vfp2h\" (UID: \"abb3d52d-bb53-47f0-9708-075b2eae0e2d\") " pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:23:41.664874 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:41.664854 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:23:41.675070 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:41.675041 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6d9fe54-0654-4f9d-bb79-0fbeec14c98f-original-pull-secret\") pod \"global-pull-secret-syncer-kpsb4\" (UID: \"a6d9fe54-0654-4f9d-bb79-0fbeec14c98f\") " pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:23:41.689682 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:41.689649 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kpsb4" Jul 06 19:23:41.698095 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:41.698071 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tr4sh\"" Jul 06 19:23:41.706004 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:41.705979 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:23:41.706136 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:41.706072 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d9llp\"" Jul 06 19:23:41.714387 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:41.714357 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lh7hd" Jul 06 19:23:43.277511 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:43.277469 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:43.280321 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:43.280294 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls\") pod \"image-registry-954dc4658-xvp8b\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:43.378029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:43.377992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:43.378236 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:43.378051 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:43.378236 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:43.378091 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-z8zqc\" (UID: \"8aa685e5-fc07-4d7b-bb90-506c9c9f34be\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" Jul 06 19:23:43.378724 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:43.378680 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc56f897-5d59-4451-a517-5bfea063ce0c-service-ca-bundle\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:43.380909 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:43.380886 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8aa685e5-fc07-4d7b-bb90-506c9c9f34be-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-z8zqc\" (UID: \"8aa685e5-fc07-4d7b-bb90-506c9c9f34be\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" Jul 06 19:23:43.381002 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:43.380886 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc56f897-5d59-4451-a517-5bfea063ce0c-metrics-certs\") pod \"router-default-68d9d48ccc-kk7pz\" (UID: \"cc56f897-5d59-4451-a517-5bfea063ce0c\") " pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:43.472464 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:43.472434 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jf8st\"" Jul 06 19:23:43.478437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:43.478409 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls\") pod \"dns-default-8j849\" (UID: \"670f9fdc-7f0b-44f5-9277-c88477bde835\") " pod="openshift-dns/dns-default-8j849" Jul 06 19:23:43.479944 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:43.479918 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:43.480806 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:43.480786 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/670f9fdc-7f0b-44f5-9277-c88477bde835-metrics-tls\") pod \"dns-default-8j849\" (UID: \"670f9fdc-7f0b-44f5-9277-c88477bde835\") " pod="openshift-dns/dns-default-8j849" Jul 06 19:23:43.488379 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:43.488354 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mv8tf\"" Jul 06 19:23:43.492300 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:43.492274 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rbhnq\"" Jul 06 19:23:43.495032 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:43.495009 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" Jul 06 19:23:43.501284 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:43.501260 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:43.615998 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:43.615972 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bhtlv\"" Jul 06 19:23:43.622283 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:43.622248 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8j849" Jul 06 19:23:44.262809 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:44.262731 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kpsb4"] Jul 06 19:23:44.333427 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:44.333337 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lh7hd"] Jul 06 19:23:44.397905 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:44.397867 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d9fe54_0654_4f9d_bb79_0fbeec14c98f.slice/crio-ccbf36b6a39b0492913ddfeaeaf96b395d79f47643e68f1ba03da5e65d6d8fe9 WatchSource:0}: Error finding container ccbf36b6a39b0492913ddfeaeaf96b395d79f47643e68f1ba03da5e65d6d8fe9: Status 404 returned error can't find the container with id ccbf36b6a39b0492913ddfeaeaf96b395d79f47643e68f1ba03da5e65d6d8fe9 Jul 06 19:23:44.398536 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:44.398409 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362f1775_4ab7_400a_b216_a22e4ca6c9fb.slice/crio-b970086c7860b9f3253207de3eeb9d59d71c1c5fbd01952c114d03e8b0f06793 WatchSource:0}: Error finding container b970086c7860b9f3253207de3eeb9d59d71c1c5fbd01952c114d03e8b0f06793: Status 404 returned error can't find the container with id b970086c7860b9f3253207de3eeb9d59d71c1c5fbd01952c114d03e8b0f06793 Jul 06 19:23:44.494103 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:44.494014 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vfp2h"] Jul 06 19:23:44.495675 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:44.495654 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8j849"] Jul 06 19:23:44.498456 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:44.498426 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb3d52d_bb53_47f0_9708_075b2eae0e2d.slice/crio-05bc1496cafbb6879846162d09fc074b164f0dd9000726bc23f0af926cf55089 WatchSource:0}: Error finding container 05bc1496cafbb6879846162d09fc074b164f0dd9000726bc23f0af926cf55089: Status 404 returned error can't find the container with id 05bc1496cafbb6879846162d09fc074b164f0dd9000726bc23f0af926cf55089 Jul 06 19:23:44.499391 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:44.499299 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670f9fdc_7f0b_44f5_9277_c88477bde835.slice/crio-196dcb35e98ac7eb6174adae673071b12faf311bac8edeb1df90ca5f121326b6 WatchSource:0}: Error finding container 196dcb35e98ac7eb6174adae673071b12faf311bac8edeb1df90ca5f121326b6: Status 404 returned error can't find the container with id 196dcb35e98ac7eb6174adae673071b12faf311bac8edeb1df90ca5f121326b6 Jul 06 19:23:44.510460 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:44.510395 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc"] Jul 06 19:23:44.514367 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:44.514338 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa685e5_fc07_4d7b_bb90_506c9c9f34be.slice/crio-08ee3662ed7e061b78ef69869c96b5625f967546f1c6d57e3b69064f50ed36ea WatchSource:0}: Error finding container 08ee3662ed7e061b78ef69869c96b5625f967546f1c6d57e3b69064f50ed36ea: Status 404 returned error can't find the container with id 08ee3662ed7e061b78ef69869c96b5625f967546f1c6d57e3b69064f50ed36ea Jul 06 19:23:44.568903 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:44.568874 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68d9d48ccc-kk7pz"] Jul 06 19:23:44.571987 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:44.571964 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc56f897_5d59_4451_a517_5bfea063ce0c.slice/crio-69321bf2a172045490a5341ea85c37d61c85187e3517f197f1b6ad202873f1e3 WatchSource:0}: Error finding container 69321bf2a172045490a5341ea85c37d61c85187e3517f197f1b6ad202873f1e3: Status 404 returned error can't find the container with id 69321bf2a172045490a5341ea85c37d61c85187e3517f197f1b6ad202873f1e3 Jul 06 19:23:44.578394 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:44.578372 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-954dc4658-xvp8b"] Jul 06 19:23:44.582163 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:44.582141 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee43e118_575d_4aa1_9276_f9f1ed3606a4.slice/crio-71d49a028cc05935109a9009e793ca3f3116e2bc5f5f2748ed6f98dce4c4dc95 WatchSource:0}: Error finding container 71d49a028cc05935109a9009e793ca3f3116e2bc5f5f2748ed6f98dce4c4dc95: Status 404 returned error can't find the container with id 71d49a028cc05935109a9009e793ca3f3116e2bc5f5f2748ed6f98dce4c4dc95 Jul 06 19:23:44.869798 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:44.869710 2578 scope.go:117] "RemoveContainer" containerID="502afffb18f0176ddb764a418fe4d2b67067ce854f00675317a02f1b8bae697b" Jul 06 19:23:45.236388 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.236258 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kpsb4" event={"ID":"a6d9fe54-0654-4f9d-bb79-0fbeec14c98f","Type":"ContainerStarted","Data":"ccbf36b6a39b0492913ddfeaeaf96b395d79f47643e68f1ba03da5e65d6d8fe9"} Jul 06 19:23:45.239749 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.239713 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vfp2h" event={"ID":"abb3d52d-bb53-47f0-9708-075b2eae0e2d","Type":"ContainerStarted","Data":"ae978ee4b12abb32ecaa7133c42783646c688c0fddd3e17525f4c2ecbd9e2f50"} Jul 06 19:23:45.239917 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.239772 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vfp2h" event={"ID":"abb3d52d-bb53-47f0-9708-075b2eae0e2d","Type":"ContainerStarted","Data":"05bc1496cafbb6879846162d09fc074b164f0dd9000726bc23f0af926cf55089"} Jul 06 19:23:45.240019 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.239998 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:23:45.243675 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.243648 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 19:23:45.243878 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.243796 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-62lvt" event={"ID":"09143401-93b0-453e-a738-9b8fa5457a81","Type":"ContainerStarted","Data":"b94be49182479afa4600df8b4a7af0fda93fff7a7eb40c5cb60ae5942290d759"} Jul 06 19:23:45.244552 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.244526 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-62lvt" Jul 06 19:23:45.250245 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.250197 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cxsxv" event={"ID":"61241f90-6bd8-4a8c-8784-02757d7fcb36","Type":"ContainerStarted","Data":"a16821bb933da55d13f3c3119c7ce5a3c35352097ce9330d9cac559e7733d44f"} Jul 06 19:23:45.250402 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.250260 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cxsxv" event={"ID":"61241f90-6bd8-4a8c-8784-02757d7fcb36","Type":"ContainerStarted","Data":"83ec5eaf30077d9f79bf3bfb09994029e1c2cc7832db66fbc93fa048b5a8112b"} Jul 06 19:23:45.253963 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.253933 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" event={"ID":"cc56f897-5d59-4451-a517-5bfea063ce0c","Type":"ContainerStarted","Data":"a86f76593538e0e8974088b492f1a66c713dc225f6a4c4e92b4e2cb64327a70a"} Jul 06 19:23:45.254048 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.253985 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" event={"ID":"cc56f897-5d59-4451-a517-5bfea063ce0c","Type":"ContainerStarted","Data":"69321bf2a172045490a5341ea85c37d61c85187e3517f197f1b6ad202873f1e3"} Jul 06 19:23:45.255785 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.255720 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lh7hd" event={"ID":"362f1775-4ab7-400a-b216-a22e4ca6c9fb","Type":"ContainerStarted","Data":"b970086c7860b9f3253207de3eeb9d59d71c1c5fbd01952c114d03e8b0f06793"} Jul 06 19:23:45.257802 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.257681 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-954dc4658-xvp8b" event={"ID":"ee43e118-575d-4aa1-9276-f9f1ed3606a4","Type":"ContainerStarted","Data":"6ab534549a4694c28874d28d9783babb6580e37cdb358428690f76011265c84e"} Jul 06 19:23:45.257802 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.257773 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-954dc4658-xvp8b" event={"ID":"ee43e118-575d-4aa1-9276-f9f1ed3606a4","Type":"ContainerStarted","Data":"71d49a028cc05935109a9009e793ca3f3116e2bc5f5f2748ed6f98dce4c4dc95"} Jul 06 19:23:45.257964 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.257927 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:23:45.260425 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.260402 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8j849" event={"ID":"670f9fdc-7f0b-44f5-9277-c88477bde835","Type":"ContainerStarted","Data":"196dcb35e98ac7eb6174adae673071b12faf311bac8edeb1df90ca5f121326b6"} Jul 06 19:23:45.261815 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.261767 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" event={"ID":"8aa685e5-fc07-4d7b-bb90-506c9c9f34be","Type":"ContainerStarted","Data":"08ee3662ed7e061b78ef69869c96b5625f967546f1c6d57e3b69064f50ed36ea"} Jul 06 19:23:45.298010 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.297353 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vfp2h" podStartSLOduration=69.297332692 podStartE2EDuration="1m9.297332692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:45.296603877 +0000 UTC m=+69.925557845" watchObservedRunningTime="2026-07-06 19:23:45.297332692 +0000 UTC m=+69.926286660" Jul 06 19:23:45.332526 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.331293 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-954dc4658-xvp8b" podStartSLOduration=68.331276793 podStartE2EDuration="1m8.331276793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:45.330993379 +0000 UTC m=+69.959947344" watchObservedRunningTime="2026-07-06 19:23:45.331276793 +0000 UTC m=+69.960230758" Jul 06 19:23:45.399839 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.398938 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-62lvt" podStartSLOduration=21.399752321 podStartE2EDuration="25.398916747s" totalImagesPullingTime="3.999164426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:45.358424302 +0000 UTC m=+69.987378278" watchObservedRunningTime="2026-07-06 19:23:45.398916747 +0000 UTC m=+70.027870714" Jul 06 19:23:45.399839 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.399578 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" podStartSLOduration=35.399565305 podStartE2EDuration="35.399565305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:45.396865541 +0000 UTC m=+70.025819511" watchObservedRunningTime="2026-07-06 19:23:45.399565305 +0000 UTC m=+70.028519272" Jul 06 19:23:45.502181 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.502102 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:45.504940 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.504913 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:45.867862 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:45.867550 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-62lvt" Jul 06 19:23:46.272912 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:46.272879 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lh7hd" event={"ID":"362f1775-4ab7-400a-b216-a22e4ca6c9fb","Type":"ContainerStarted","Data":"cf37e546ae95c5f222227d2570fd267bf50a66d09e3afea0b941cb1b48e90f26"} Jul 06 19:23:46.273198 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:46.272945 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lh7hd" event={"ID":"362f1775-4ab7-400a-b216-a22e4ca6c9fb","Type":"ContainerStarted","Data":"b29e2ec67a816842ea356eb2f8a4cf80804b4923f48234c5e35d409aa5873261"} Jul 06 19:23:46.273526 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:46.273470 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:46.275153 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:46.275129 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-68d9d48ccc-kk7pz" Jul 06 19:23:46.297101 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:46.296674 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lh7hd" podStartSLOduration=68.990956792 podStartE2EDuration="1m10.296651914s" totalImagesPullingTime="1.305695122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:46.294980202 +0000 UTC m=+70.923934169" watchObservedRunningTime="2026-07-06 19:23:46.296651914 +0000 UTC m=+70.925605883" Jul 06 19:23:51.290236 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.290201 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-mjdwf"] Jul 06 19:23:51.313607 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.313573 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cxsxv" event={"ID":"61241f90-6bd8-4a8c-8784-02757d7fcb36","Type":"ContainerStarted","Data":"bd796aed81725808448a0ad611e4638882b01490b553ec95889885115d4bef86"} Jul 06 19:23:51.313607 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.313612 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8j849" event={"ID":"670f9fdc-7f0b-44f5-9277-c88477bde835","Type":"ContainerStarted","Data":"5da126e66f375e94c626d60ad6e148a8bd379c39c6a17565e228f543b5876a6b"} Jul 06 19:23:51.313849 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.313675 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-mjdwf" Jul 06 19:23:51.313849 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.313675 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8j849" Jul 06 19:23:51.313849 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.313735 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8j849" event={"ID":"670f9fdc-7f0b-44f5-9277-c88477bde835","Type":"ContainerStarted","Data":"43b46445dce3008f06277a280cbd11ae2524db4ceed19c8973b9174713268de3"} Jul 06 19:23:51.313849 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.313761 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" event={"ID":"8aa685e5-fc07-4d7b-bb90-506c9c9f34be","Type":"ContainerStarted","Data":"2650ada11d21f832d3d28d2ae9395d78e11d6511e9c74e0bb463c2037379d1a6"} Jul 06 19:23:51.313849 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.313780 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kpsb4" event={"ID":"a6d9fe54-0654-4f9d-bb79-0fbeec14c98f","Type":"ContainerStarted","Data":"b306a88799ae2663e1f7519df3b18466ddf4f6430af0d5a4f7704c758862daf6"} Jul 06 19:23:51.316429 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.316408 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 06 19:23:51.316552 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.316415 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-64g6w\"" Jul 06 19:23:51.316552 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.316454 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 06 19:23:51.326008 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.325980 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-mjdwf"] Jul 06 19:23:51.326274 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.326232 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cxsxv" podStartSLOduration=18.100256438 podStartE2EDuration="28.326212602s" totalImagesPullingTime="10.225956164s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:23:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:51.322434867 +0000 UTC m=+75.951388832" watchObservedRunningTime="2026-07-06 19:23:51.326212602 +0000 UTC m=+75.955166568" Jul 06 19:23:51.350608 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.350203 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bdf0c25-68fd-43aa-b54b-baf04fdfe2a0-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-mjdwf\" (UID: \"5bdf0c25-68fd-43aa-b54b-baf04fdfe2a0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-mjdwf" Jul 06 19:23:51.350608 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.350273 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5bdf0c25-68fd-43aa-b54b-baf04fdfe2a0-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-mjdwf\" (UID: \"5bdf0c25-68fd-43aa-b54b-baf04fdfe2a0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-mjdwf" Jul 06 19:23:51.363956 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.363892 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-z8zqc" podStartSLOduration=35.508235689 podStartE2EDuration="41.363873421s" totalImagesPullingTime="5.855637732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:51.362930394 +0000 UTC m=+75.991884361" watchObservedRunningTime="2026-07-06 19:23:51.363873421 +0000 UTC m=+75.992827387" Jul 06 19:23:51.396031 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.395992 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-cg86p"] Jul 06 19:23:51.407681 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.407628 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kpsb4" podStartSLOduration=68.096693728 podStartE2EDuration="1m14.40761274s" totalImagesPullingTime="6.310919012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:51.405254014 +0000 UTC m=+76.034207979" watchObservedRunningTime="2026-07-06 19:23:51.40761274 +0000 UTC m=+76.036566700" Jul 06 19:23:51.414025 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.413983 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-hstnz"] Jul 06 19:23:51.414158 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.414119 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-cg86p" Jul 06 19:23:51.417112 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.416999 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-lxd7r\"" Jul 06 19:23:51.417687 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.417662 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 19:23:51.417911 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.417893 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 19:23:51.433254 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.433220 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-hstnz"] Jul 06 19:23:51.433254 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.433259 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-cg86p"] Jul 06 19:23:51.433466 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.433351 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-hstnz" Jul 06 19:23:51.441304 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.439893 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-q9ntk\"" Jul 06 19:23:51.443998 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.443974 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 06 19:23:51.450673 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.450647 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5bbb14fe-9b7b-45d8-9401-3ff9468879d0-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-hstnz\" (UID: \"5bbb14fe-9b7b-45d8-9401-3ff9468879d0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-hstnz" Jul 06 19:23:51.450842 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.450719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bdf0c25-68fd-43aa-b54b-baf04fdfe2a0-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-mjdwf\" (UID: \"5bdf0c25-68fd-43aa-b54b-baf04fdfe2a0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-mjdwf" Jul 06 19:23:51.450842 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.450788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5bdf0c25-68fd-43aa-b54b-baf04fdfe2a0-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-mjdwf\" (UID: \"5bdf0c25-68fd-43aa-b54b-baf04fdfe2a0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-mjdwf" Jul 06 19:23:51.450989 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.450850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czkvr\" (UniqueName: \"kubernetes.io/projected/1bb7129a-88cf-4c6e-b47b-9315e0721dce-kube-api-access-czkvr\") pod \"downloads-685bfb5f64-cg86p\" (UID: \"1bb7129a-88cf-4c6e-b47b-9315e0721dce\") " pod="openshift-console/downloads-685bfb5f64-cg86p" Jul 06 19:23:51.451417 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.451399 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5bdf0c25-68fd-43aa-b54b-baf04fdfe2a0-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-mjdwf\" (UID: \"5bdf0c25-68fd-43aa-b54b-baf04fdfe2a0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-mjdwf" Jul 06 19:23:51.453048 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.453020 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bdf0c25-68fd-43aa-b54b-baf04fdfe2a0-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-mjdwf\" (UID: \"5bdf0c25-68fd-43aa-b54b-baf04fdfe2a0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-mjdwf" Jul 06 19:23:51.506647 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.506590 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8j849" podStartSLOduration=34.636736997 podStartE2EDuration="40.5065737s" totalImagesPullingTime="5.869836703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:51.505959763 +0000 UTC m=+76.134913729" watchObservedRunningTime="2026-07-06 19:23:51.5065737 +0000 UTC m=+76.135527667" Jul 06 19:23:51.552383 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.552351 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czkvr\" (UniqueName: \"kubernetes.io/projected/1bb7129a-88cf-4c6e-b47b-9315e0721dce-kube-api-access-czkvr\") pod \"downloads-685bfb5f64-cg86p\" (UID: \"1bb7129a-88cf-4c6e-b47b-9315e0721dce\") " pod="openshift-console/downloads-685bfb5f64-cg86p" Jul 06 19:23:51.552592 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.552405 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5bbb14fe-9b7b-45d8-9401-3ff9468879d0-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-hstnz\" (UID: \"5bbb14fe-9b7b-45d8-9401-3ff9468879d0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-hstnz" Jul 06 19:23:51.555093 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.555064 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5bbb14fe-9b7b-45d8-9401-3ff9468879d0-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-hstnz\" (UID: \"5bbb14fe-9b7b-45d8-9401-3ff9468879d0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-hstnz" Jul 06 19:23:51.577075 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.577043 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czkvr\" (UniqueName: \"kubernetes.io/projected/1bb7129a-88cf-4c6e-b47b-9315e0721dce-kube-api-access-czkvr\") pod \"downloads-685bfb5f64-cg86p\" (UID: \"1bb7129a-88cf-4c6e-b47b-9315e0721dce\") " pod="openshift-console/downloads-685bfb5f64-cg86p" Jul 06 19:23:51.622852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.622804 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-mjdwf" Jul 06 19:23:51.726370 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.726336 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-cg86p" Jul 06 19:23:51.744451 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.744417 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-hstnz" Jul 06 19:23:51.840330 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:51.840265 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bdf0c25_68fd_43aa_b54b_baf04fdfe2a0.slice/crio-63c1051a4def31aab1941e0f4d451bcab7e487c38164a3b9921d240b671edb12 WatchSource:0}: Error finding container 63c1051a4def31aab1941e0f4d451bcab7e487c38164a3b9921d240b671edb12: Status 404 returned error can't find the container with id 63c1051a4def31aab1941e0f4d451bcab7e487c38164a3b9921d240b671edb12 Jul 06 19:23:51.840873 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:51.840836 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-mjdwf"] Jul 06 19:23:52.000636 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:52.000609 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-cg86p"] Jul 06 19:23:52.002036 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:52.002009 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb7129a_88cf_4c6e_b47b_9315e0721dce.slice/crio-811d5075421a603161b0078cc37864a8ba57881d3c1120e87782b0211fb4dc91 WatchSource:0}: Error finding container 811d5075421a603161b0078cc37864a8ba57881d3c1120e87782b0211fb4dc91: Status 404 returned error can't find the container with id 811d5075421a603161b0078cc37864a8ba57881d3c1120e87782b0211fb4dc91 Jul 06 19:23:52.016934 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:52.016903 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-hstnz"] Jul 06 19:23:52.023073 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:23:52.020715 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbb14fe_9b7b_45d8_9401_3ff9468879d0.slice/crio-9501b39835c7e0bc0fd4b7b96c847e3be9b1d472f670cf6b1c468967669f8433 WatchSource:0}: Error finding container 9501b39835c7e0bc0fd4b7b96c847e3be9b1d472f670cf6b1c468967669f8433: Status 404 returned error can't find the container with id 9501b39835c7e0bc0fd4b7b96c847e3be9b1d472f670cf6b1c468967669f8433 Jul 06 19:23:52.306005 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:52.305968 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-cg86p" event={"ID":"1bb7129a-88cf-4c6e-b47b-9315e0721dce","Type":"ContainerStarted","Data":"811d5075421a603161b0078cc37864a8ba57881d3c1120e87782b0211fb4dc91"} Jul 06 19:23:52.307194 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:52.307161 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-mjdwf" event={"ID":"5bdf0c25-68fd-43aa-b54b-baf04fdfe2a0","Type":"ContainerStarted","Data":"63c1051a4def31aab1941e0f4d451bcab7e487c38164a3b9921d240b671edb12"} Jul 06 19:23:52.308393 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:52.308369 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-hstnz" event={"ID":"5bbb14fe-9b7b-45d8-9401-3ff9468879d0","Type":"ContainerStarted","Data":"9501b39835c7e0bc0fd4b7b96c847e3be9b1d472f670cf6b1c468967669f8433"} Jul 06 19:23:54.320582 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:54.320517 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-mjdwf" event={"ID":"5bdf0c25-68fd-43aa-b54b-baf04fdfe2a0","Type":"ContainerStarted","Data":"54f357b91af48c63e98365792a0cc1f28624b69d34da57355c8ad0065128a3fa"} Jul 06 19:23:54.322367 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:54.322341 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-hstnz" event={"ID":"5bbb14fe-9b7b-45d8-9401-3ff9468879d0","Type":"ContainerStarted","Data":"d28c19cc49a37f3cd687d7f80442e7352cc90b952e94261a0f513fad805f1ead"} Jul 06 19:23:54.322509 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:54.322395 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-hstnz" Jul 06 19:23:54.327597 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:54.327571 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-hstnz" Jul 06 19:23:54.351812 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:54.351707 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-mjdwf" podStartSLOduration=1.081839222 podStartE2EDuration="3.351672924s" totalImagesPullingTime="2.269833702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:54.351229339 +0000 UTC m=+78.980183304" watchObservedRunningTime="2026-07-06 19:23:54.351672924 +0000 UTC m=+78.980626918" Jul 06 19:23:54.387539 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:23:54.387472 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-hstnz" podStartSLOduration=1.3016512040000001 podStartE2EDuration="3.387448123s" totalImagesPullingTime="2.085796919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:54.385730702 +0000 UTC m=+79.014684669" watchObservedRunningTime="2026-07-06 19:23:54.387448123 +0000 UTC m=+79.016402087" Jul 06 19:24:02.317527 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.317399 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8j849" Jul 06 19:24:02.641653 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.641570 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dsgzx"] Jul 06 19:24:02.645610 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.645584 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.648105 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.648086 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 19:24:02.648297 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.648258 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 19:24:02.648422 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.648149 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 19:24:02.648500 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.648477 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 19:24:02.648744 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.648724 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4jjw4\"" Jul 06 19:24:02.744112 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.744062 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7611af4b-652f-4386-ae4a-39d0d802e146-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.744295 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.744125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7611af4b-652f-4386-ae4a-39d0d802e146-node-exporter-tls\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.744295 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.744210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7611af4b-652f-4386-ae4a-39d0d802e146-node-exporter-wtmp\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.744295 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.744263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7611af4b-652f-4386-ae4a-39d0d802e146-node-exporter-textfile\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.744461 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.744299 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7611af4b-652f-4386-ae4a-39d0d802e146-sys\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.744461 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.744327 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7611af4b-652f-4386-ae4a-39d0d802e146-node-exporter-accelerators-collector-config\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.744461 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.744367 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpfbr\" (UniqueName: \"kubernetes.io/projected/7611af4b-652f-4386-ae4a-39d0d802e146-kube-api-access-zpfbr\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.744461 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.744434 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7611af4b-652f-4386-ae4a-39d0d802e146-metrics-client-ca\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.744663 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.744483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7611af4b-652f-4386-ae4a-39d0d802e146-root\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.845369 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.845305 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7611af4b-652f-4386-ae4a-39d0d802e146-root\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.845369 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.845331 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7611af4b-652f-4386-ae4a-39d0d802e146-root\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.845589 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.845425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7611af4b-652f-4386-ae4a-39d0d802e146-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.845589 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.845465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7611af4b-652f-4386-ae4a-39d0d802e146-node-exporter-tls\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.845589 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.845495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7611af4b-652f-4386-ae4a-39d0d802e146-node-exporter-wtmp\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.845589 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.845538 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7611af4b-652f-4386-ae4a-39d0d802e146-node-exporter-textfile\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.845589 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.845576 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7611af4b-652f-4386-ae4a-39d0d802e146-sys\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.845922 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.845606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7611af4b-652f-4386-ae4a-39d0d802e146-node-exporter-accelerators-collector-config\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.845922 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.845640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpfbr\" (UniqueName: \"kubernetes.io/projected/7611af4b-652f-4386-ae4a-39d0d802e146-kube-api-access-zpfbr\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.845922 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.845688 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7611af4b-652f-4386-ae4a-39d0d802e146-metrics-client-ca\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.846383 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.846198 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7611af4b-652f-4386-ae4a-39d0d802e146-node-exporter-textfile\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.846383 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.846283 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7611af4b-652f-4386-ae4a-39d0d802e146-sys\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.846383 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.846340 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7611af4b-652f-4386-ae4a-39d0d802e146-node-exporter-wtmp\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.846584 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.846437 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7611af4b-652f-4386-ae4a-39d0d802e146-metrics-client-ca\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.846584 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.846508 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7611af4b-652f-4386-ae4a-39d0d802e146-node-exporter-accelerators-collector-config\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.848678 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.848598 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7611af4b-652f-4386-ae4a-39d0d802e146-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.849355 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.849330 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7611af4b-652f-4386-ae4a-39d0d802e146-node-exporter-tls\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.871201 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.871138 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpfbr\" (UniqueName: \"kubernetes.io/projected/7611af4b-652f-4386-ae4a-39d0d802e146-kube-api-access-zpfbr\") pod \"node-exporter-dsgzx\" (UID: \"7611af4b-652f-4386-ae4a-39d0d802e146\") " pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:02.958871 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:02.958791 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dsgzx" Jul 06 19:24:03.485262 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:03.485205 2578 patch_prober.go:28] interesting pod/image-registry-954dc4658-xvp8b container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 19:24:03.485730 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:03.485279 2578 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-954dc4658-xvp8b" podUID="ee43e118-575d-4aa1-9276-f9f1ed3606a4" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:24:06.277947 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:06.277917 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:24:10.165003 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:24:10.164954 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7611af4b_652f_4386_ae4a_39d0d802e146.slice/crio-5c1f0e072047fa70aed8fe8f41368167f738cf6815cc923406f6dc4a6db8f85e WatchSource:0}: Error finding container 5c1f0e072047fa70aed8fe8f41368167f738cf6815cc923406f6dc4a6db8f85e: Status 404 returned error can't find the container with id 5c1f0e072047fa70aed8fe8f41368167f738cf6815cc923406f6dc4a6db8f85e Jul 06 19:24:10.404043 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:10.404004 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dsgzx" event={"ID":"7611af4b-652f-4386-ae4a-39d0d802e146","Type":"ContainerStarted","Data":"5c1f0e072047fa70aed8fe8f41368167f738cf6815cc923406f6dc4a6db8f85e"} Jul 06 19:24:11.412404 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:11.412185 2578 generic.go:356] "Generic (PLEG): container finished" podID="7611af4b-652f-4386-ae4a-39d0d802e146" containerID="b16903abc303d2b087a6c4cc44e00888aaa6bc8c6ddda7ade40ccaa5e78992a0" exitCode=0 Jul 06 19:24:11.412404 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:11.412263 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dsgzx" event={"ID":"7611af4b-652f-4386-ae4a-39d0d802e146","Type":"ContainerDied","Data":"b16903abc303d2b087a6c4cc44e00888aaa6bc8c6ddda7ade40ccaa5e78992a0"} Jul 06 19:24:11.414770 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:11.414740 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-cg86p" event={"ID":"1bb7129a-88cf-4c6e-b47b-9315e0721dce","Type":"ContainerStarted","Data":"952133a7772cb89878a9c35fdcac5199979fe34fde26ba9034975ccdfa736759"} Jul 06 19:24:11.414909 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:11.414805 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-cg86p" Jul 06 19:24:11.436335 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:11.436302 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-cg86p" Jul 06 19:24:11.517066 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:11.517006 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-cg86p" podStartSLOduration=1.9948056699999999 podStartE2EDuration="20.516984702s" totalImagesPullingTime="18.522179032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:11.515988066 +0000 UTC m=+96.144942033" watchObservedRunningTime="2026-07-06 19:24:11.516984702 +0000 UTC m=+96.145938669" Jul 06 19:24:12.422975 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:12.422933 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dsgzx" event={"ID":"7611af4b-652f-4386-ae4a-39d0d802e146","Type":"ContainerStarted","Data":"10eb53ff79a1c2303854cbeb07a89364b555740ca7f45ebcac53a87d012763bf"} Jul 06 19:24:12.423449 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:12.423006 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dsgzx" event={"ID":"7611af4b-652f-4386-ae4a-39d0d802e146","Type":"ContainerStarted","Data":"27230e9fa4ee0ba476906dffcf6f25828af32dc2ffd996a5dbd7b2de286786cf"} Jul 06 19:24:12.454998 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:12.454937 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dsgzx" podStartSLOduration=9.404553109 podStartE2EDuration="10.454914003s" totalImagesPullingTime="970.038465ms" totalInitContainerRuntime="80.322429ms" isStatefulPod=true podCreationTimestamp="2026-07-06 19:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:12.453253177 +0000 UTC m=+97.082207166" watchObservedRunningTime="2026-07-06 19:24:12.454914003 +0000 UTC m=+97.083867993" Jul 06 19:24:13.734221 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:13.734183 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-954dc4658-xvp8b"] Jul 06 19:24:16.276190 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:16.276159 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vfp2h" Jul 06 19:24:38.760506 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:38.760454 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-954dc4658-xvp8b" podUID="ee43e118-575d-4aa1-9276-f9f1ed3606a4" containerName="registry" containerID="cri-o://6ab534549a4694c28874d28d9783babb6580e37cdb358428690f76011265c84e" gracePeriod=30 Jul 06 19:24:39.038547 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.038524 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:24:39.062192 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.062160 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrmb8\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-kube-api-access-vrmb8\") pod \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " Jul 06 19:24:39.062369 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.062223 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee43e118-575d-4aa1-9276-f9f1ed3606a4-ca-trust-extracted\") pod \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " Jul 06 19:24:39.062369 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.062293 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee43e118-575d-4aa1-9276-f9f1ed3606a4-trusted-ca\") pod \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " Jul 06 19:24:39.062369 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.062325 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls\") pod \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " Jul 06 19:24:39.062369 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.062360 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee43e118-575d-4aa1-9276-f9f1ed3606a4-image-registry-private-configuration\") pod \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " Jul 06 19:24:39.062594 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.062393 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-bound-sa-token\") pod \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " Jul 06 19:24:39.063162 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.063125 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee43e118-575d-4aa1-9276-f9f1ed3606a4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ee43e118-575d-4aa1-9276-f9f1ed3606a4" (UID: "ee43e118-575d-4aa1-9276-f9f1ed3606a4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:24:39.065243 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.065186 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ee43e118-575d-4aa1-9276-f9f1ed3606a4" (UID: "ee43e118-575d-4aa1-9276-f9f1ed3606a4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:24:39.065368 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.065316 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-kube-api-access-vrmb8" (OuterVolumeSpecName: "kube-api-access-vrmb8") pod "ee43e118-575d-4aa1-9276-f9f1ed3606a4" (UID: "ee43e118-575d-4aa1-9276-f9f1ed3606a4"). InnerVolumeSpecName "kube-api-access-vrmb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:24:39.065430 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.065398 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee43e118-575d-4aa1-9276-f9f1ed3606a4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ee43e118-575d-4aa1-9276-f9f1ed3606a4" (UID: "ee43e118-575d-4aa1-9276-f9f1ed3606a4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:24:39.065735 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.065715 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ee43e118-575d-4aa1-9276-f9f1ed3606a4" (UID: "ee43e118-575d-4aa1-9276-f9f1ed3606a4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:24:39.071085 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.071062 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee43e118-575d-4aa1-9276-f9f1ed3606a4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ee43e118-575d-4aa1-9276-f9f1ed3606a4" (UID: "ee43e118-575d-4aa1-9276-f9f1ed3606a4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:24:39.163001 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.162969 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee43e118-575d-4aa1-9276-f9f1ed3606a4-installation-pull-secrets\") pod \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " Jul 06 19:24:39.163001 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.163003 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-certificates\") pod \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\" (UID: \"ee43e118-575d-4aa1-9276-f9f1ed3606a4\") " Jul 06 19:24:39.163259 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.163146 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee43e118-575d-4aa1-9276-f9f1ed3606a4-ca-trust-extracted\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:24:39.163259 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.163156 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee43e118-575d-4aa1-9276-f9f1ed3606a4-trusted-ca\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:24:39.163259 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.163164 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-tls\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:24:39.163259 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.163172 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee43e118-575d-4aa1-9276-f9f1ed3606a4-image-registry-private-configuration\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:24:39.163259 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.163180 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-bound-sa-token\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:24:39.163259 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.163189 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrmb8\" (UniqueName: \"kubernetes.io/projected/ee43e118-575d-4aa1-9276-f9f1ed3606a4-kube-api-access-vrmb8\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:24:39.163537 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.163465 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ee43e118-575d-4aa1-9276-f9f1ed3606a4" (UID: "ee43e118-575d-4aa1-9276-f9f1ed3606a4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:24:39.165075 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.165052 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee43e118-575d-4aa1-9276-f9f1ed3606a4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ee43e118-575d-4aa1-9276-f9f1ed3606a4" (UID: "ee43e118-575d-4aa1-9276-f9f1ed3606a4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:24:39.264220 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.264182 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee43e118-575d-4aa1-9276-f9f1ed3606a4-installation-pull-secrets\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:24:39.264220 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.264218 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee43e118-575d-4aa1-9276-f9f1ed3606a4-registry-certificates\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:24:39.543300 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.543268 2578 generic.go:356] "Generic (PLEG): container finished" podID="ee43e118-575d-4aa1-9276-f9f1ed3606a4" containerID="6ab534549a4694c28874d28d9783babb6580e37cdb358428690f76011265c84e" exitCode=0 Jul 06 19:24:39.543475 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.543316 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-954dc4658-xvp8b" event={"ID":"ee43e118-575d-4aa1-9276-f9f1ed3606a4","Type":"ContainerDied","Data":"6ab534549a4694c28874d28d9783babb6580e37cdb358428690f76011265c84e"} Jul 06 19:24:39.543475 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.543328 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-954dc4658-xvp8b" Jul 06 19:24:39.543475 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.543377 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-954dc4658-xvp8b" event={"ID":"ee43e118-575d-4aa1-9276-f9f1ed3606a4","Type":"ContainerDied","Data":"71d49a028cc05935109a9009e793ca3f3116e2bc5f5f2748ed6f98dce4c4dc95"} Jul 06 19:24:39.543475 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.543397 2578 scope.go:117] "RemoveContainer" containerID="6ab534549a4694c28874d28d9783babb6580e37cdb358428690f76011265c84e" Jul 06 19:24:39.551809 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.551790 2578 scope.go:117] "RemoveContainer" containerID="6ab534549a4694c28874d28d9783babb6580e37cdb358428690f76011265c84e" Jul 06 19:24:39.552086 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:24:39.552061 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab534549a4694c28874d28d9783babb6580e37cdb358428690f76011265c84e\": container with ID starting with 6ab534549a4694c28874d28d9783babb6580e37cdb358428690f76011265c84e not found: ID does not exist" containerID="6ab534549a4694c28874d28d9783babb6580e37cdb358428690f76011265c84e" Jul 06 19:24:39.552148 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.552094 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab534549a4694c28874d28d9783babb6580e37cdb358428690f76011265c84e"} err="failed to get container status \"6ab534549a4694c28874d28d9783babb6580e37cdb358428690f76011265c84e\": rpc error: code = NotFound desc = could not find container \"6ab534549a4694c28874d28d9783babb6580e37cdb358428690f76011265c84e\": container with ID starting with 6ab534549a4694c28874d28d9783babb6580e37cdb358428690f76011265c84e not found: ID does not exist" Jul 06 19:24:39.567326 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.567300 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-954dc4658-xvp8b"] Jul 06 19:24:39.572048 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.572027 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-954dc4658-xvp8b"] Jul 06 19:24:39.874398 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:39.874365 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee43e118-575d-4aa1-9276-f9f1ed3606a4" path="/var/lib/kubelet/pods/ee43e118-575d-4aa1-9276-f9f1ed3606a4/volumes" Jul 06 19:24:43.562801 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:43.562766 2578 generic.go:356] "Generic (PLEG): container finished" podID="7bf04227-6bc5-4797-a189-f1c12841bfc4" containerID="b8945a00701998363c73c1f26073c3464e6ee03e72e7b4a6bfaf1cca94d36605" exitCode=0 Jul 06 19:24:43.563187 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:43.562840 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" event={"ID":"7bf04227-6bc5-4797-a189-f1c12841bfc4","Type":"ContainerDied","Data":"b8945a00701998363c73c1f26073c3464e6ee03e72e7b4a6bfaf1cca94d36605"} Jul 06 19:24:43.563231 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:43.563195 2578 scope.go:117] "RemoveContainer" containerID="b8945a00701998363c73c1f26073c3464e6ee03e72e7b4a6bfaf1cca94d36605" Jul 06 19:24:44.568995 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:44.568956 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xgcrt" event={"ID":"7bf04227-6bc5-4797-a189-f1c12841bfc4","Type":"ContainerStarted","Data":"5fa41db9b63a5dbea3e5a7e04a2e8db022a5ca4605af26c0044c96aaa6c2cdcb"} Jul 06 19:24:53.348021 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:53.347983 2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" podUID="0f101d66-483d-43f5-89f7-cb8dcc39f131" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 06 19:24:53.607128 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:53.607052 2578 generic.go:356] "Generic (PLEG): container finished" podID="f02ed85e-784b-4f19-bd40-c79ebd77a8a9" containerID="fe5cef041480ec2372362f8c10a0b6a266180b3fcc072378aa0a22ebe531626d" exitCode=0 Jul 06 19:24:53.607128 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:53.607114 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-rhbbf" event={"ID":"f02ed85e-784b-4f19-bd40-c79ebd77a8a9","Type":"ContainerDied","Data":"fe5cef041480ec2372362f8c10a0b6a266180b3fcc072378aa0a22ebe531626d"} Jul 06 19:24:53.607444 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:53.607431 2578 scope.go:117] "RemoveContainer" containerID="fe5cef041480ec2372362f8c10a0b6a266180b3fcc072378aa0a22ebe531626d" Jul 06 19:24:54.615067 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:54.615035 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-rhbbf" event={"ID":"f02ed85e-784b-4f19-bd40-c79ebd77a8a9","Type":"ContainerStarted","Data":"dbf0648679dbb8fff65eb7db154afe976615ab79d2f5b6cf6715e023fa2879f2"} Jul 06 19:24:58.639161 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:58.639130 2578 generic.go:356] "Generic (PLEG): container finished" podID="fccf3d5c-efcb-456c-8dc9-9aa7191184d4" containerID="59156710fdcea168d54b0102301834972a6c4d895b7be1837ab3dedcbe677c46" exitCode=0 Jul 06 19:24:58.639589 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:58.639206 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-bx5pn" event={"ID":"fccf3d5c-efcb-456c-8dc9-9aa7191184d4","Type":"ContainerDied","Data":"59156710fdcea168d54b0102301834972a6c4d895b7be1837ab3dedcbe677c46"} Jul 06 19:24:58.639589 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:58.639522 2578 scope.go:117] "RemoveContainer" containerID="59156710fdcea168d54b0102301834972a6c4d895b7be1837ab3dedcbe677c46" Jul 06 19:24:59.645388 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:24:59.645355 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-bx5pn" event={"ID":"fccf3d5c-efcb-456c-8dc9-9aa7191184d4","Type":"ContainerStarted","Data":"729c69f47a316efa3734f68016f18ccad63ce656c165df3c59b549306763ffba"} Jul 06 19:25:03.348024 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:25:03.347984 2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" podUID="0f101d66-483d-43f5-89f7-cb8dcc39f131" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 06 19:25:13.348447 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:25:13.348407 2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" podUID="0f101d66-483d-43f5-89f7-cb8dcc39f131" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 06 19:25:13.348852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:25:13.348492 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" Jul 06 19:25:13.349025 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:25:13.349007 2578 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"a8350c8e77493dee37f29bd43a501c4140051bc4a639105edae5bf979221befb"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 06 19:25:13.349063 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:25:13.349048 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" podUID="0f101d66-483d-43f5-89f7-cb8dcc39f131" containerName="service-proxy" containerID="cri-o://a8350c8e77493dee37f29bd43a501c4140051bc4a639105edae5bf979221befb" gracePeriod=30 Jul 06 19:25:13.706552 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:25:13.706464 2578 generic.go:356] "Generic (PLEG): container finished" podID="0f101d66-483d-43f5-89f7-cb8dcc39f131" containerID="a8350c8e77493dee37f29bd43a501c4140051bc4a639105edae5bf979221befb" exitCode=2 Jul 06 19:25:13.706552 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:25:13.706504 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" event={"ID":"0f101d66-483d-43f5-89f7-cb8dcc39f131","Type":"ContainerDied","Data":"a8350c8e77493dee37f29bd43a501c4140051bc4a639105edae5bf979221befb"} Jul 06 19:25:13.706552 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:25:13.706550 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-dcc4d7fbf-cz7fw" event={"ID":"0f101d66-483d-43f5-89f7-cb8dcc39f131","Type":"ContainerStarted","Data":"c357482b2169d500363d584dea1a8af697f1f03ac3c9004b8150bbd4983b50bf"} Jul 06 19:27:35.764014 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:27:35.763988 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 19:27:35.768717 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:27:35.768680 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 19:27:35.783925 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:27:35.783899 2578 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 19:28:18.802914 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.802875 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8"] Jul 06 19:28:18.803360 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.803206 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee43e118-575d-4aa1-9276-f9f1ed3606a4" containerName="registry" Jul 06 19:28:18.803360 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.803218 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee43e118-575d-4aa1-9276-f9f1ed3606a4" containerName="registry" Jul 06 19:28:18.803360 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.803275 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee43e118-575d-4aa1-9276-f9f1ed3606a4" containerName="registry" Jul 06 19:28:18.805187 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.805170 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:28:18.807817 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.807797 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 19:28:18.808149 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.808136 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 19:28:18.808316 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.808300 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 19:28:18.808494 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.808475 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zpx5b\"" Jul 06 19:28:18.809152 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.809129 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 19:28:18.816188 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.816170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1f2d36b-4239-45fa-a6d7-85c316bd426e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7865995f8c-mwbc8\" (UID: \"f1f2d36b-4239-45fa-a6d7-85c316bd426e\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:28:18.816267 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.816204 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f1f2d36b-4239-45fa-a6d7-85c316bd426e-manifests\") pod \"opendatahub-operator-controller-manager-7865995f8c-mwbc8\" (UID: \"f1f2d36b-4239-45fa-a6d7-85c316bd426e\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:28:18.816267 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.816222 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1f2d36b-4239-45fa-a6d7-85c316bd426e-tmp\") pod \"opendatahub-operator-controller-manager-7865995f8c-mwbc8\" (UID: \"f1f2d36b-4239-45fa-a6d7-85c316bd426e\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:28:18.816267 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.816246 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f1f2d36b-4239-45fa-a6d7-85c316bd426e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7865995f8c-mwbc8\" (UID: \"f1f2d36b-4239-45fa-a6d7-85c316bd426e\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:28:18.816373 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.816294 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lkpq\" (UniqueName: \"kubernetes.io/projected/f1f2d36b-4239-45fa-a6d7-85c316bd426e-kube-api-access-4lkpq\") pod \"opendatahub-operator-controller-manager-7865995f8c-mwbc8\" (UID: \"f1f2d36b-4239-45fa-a6d7-85c316bd426e\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:28:18.845410 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.845385 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8"] Jul 06 19:28:18.916844 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.916815 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lkpq\" (UniqueName: \"kubernetes.io/projected/f1f2d36b-4239-45fa-a6d7-85c316bd426e-kube-api-access-4lkpq\") pod \"opendatahub-operator-controller-manager-7865995f8c-mwbc8\" (UID: \"f1f2d36b-4239-45fa-a6d7-85c316bd426e\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:28:18.917033 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.916876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1f2d36b-4239-45fa-a6d7-85c316bd426e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7865995f8c-mwbc8\" (UID: \"f1f2d36b-4239-45fa-a6d7-85c316bd426e\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:28:18.917033 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.916932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f1f2d36b-4239-45fa-a6d7-85c316bd426e-manifests\") pod \"opendatahub-operator-controller-manager-7865995f8c-mwbc8\" (UID: \"f1f2d36b-4239-45fa-a6d7-85c316bd426e\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:28:18.917033 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.916953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1f2d36b-4239-45fa-a6d7-85c316bd426e-tmp\") pod \"opendatahub-operator-controller-manager-7865995f8c-mwbc8\" (UID: \"f1f2d36b-4239-45fa-a6d7-85c316bd426e\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:28:18.917186 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.917088 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f1f2d36b-4239-45fa-a6d7-85c316bd426e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7865995f8c-mwbc8\" (UID: \"f1f2d36b-4239-45fa-a6d7-85c316bd426e\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:28:18.917448 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.917420 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f1f2d36b-4239-45fa-a6d7-85c316bd426e-manifests\") pod \"opendatahub-operator-controller-manager-7865995f8c-mwbc8\" (UID: \"f1f2d36b-4239-45fa-a6d7-85c316bd426e\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:28:18.919187 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.919164 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1f2d36b-4239-45fa-a6d7-85c316bd426e-tmp\") pod \"opendatahub-operator-controller-manager-7865995f8c-mwbc8\" (UID: \"f1f2d36b-4239-45fa-a6d7-85c316bd426e\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:28:18.919374 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.919354 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f1f2d36b-4239-45fa-a6d7-85c316bd426e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7865995f8c-mwbc8\" (UID: \"f1f2d36b-4239-45fa-a6d7-85c316bd426e\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:28:18.919435 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.919394 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1f2d36b-4239-45fa-a6d7-85c316bd426e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7865995f8c-mwbc8\" (UID: \"f1f2d36b-4239-45fa-a6d7-85c316bd426e\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:28:18.932503 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:18.932481 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lkpq\" (UniqueName: \"kubernetes.io/projected/f1f2d36b-4239-45fa-a6d7-85c316bd426e-kube-api-access-4lkpq\") pod \"opendatahub-operator-controller-manager-7865995f8c-mwbc8\" (UID: \"f1f2d36b-4239-45fa-a6d7-85c316bd426e\") " pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:28:19.115157 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:19.115060 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:28:19.281605 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:19.281571 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8"] Jul 06 19:28:19.283932 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:28:19.283906 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f2d36b_4239_45fa_a6d7_85c316bd426e.slice/crio-eb453a6e592ebaa8e96bc8141a3ab284f8f1fd630a4f09535d71ef92a8b54ecf WatchSource:0}: Error finding container eb453a6e592ebaa8e96bc8141a3ab284f8f1fd630a4f09535d71ef92a8b54ecf: Status 404 returned error can't find the container with id eb453a6e592ebaa8e96bc8141a3ab284f8f1fd630a4f09535d71ef92a8b54ecf Jul 06 19:28:19.285650 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:19.285635 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:28:19.448241 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:19.448141 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" event={"ID":"f1f2d36b-4239-45fa-a6d7-85c316bd426e","Type":"ContainerStarted","Data":"eb453a6e592ebaa8e96bc8141a3ab284f8f1fd630a4f09535d71ef92a8b54ecf"} Jul 06 19:28:22.462761 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:22.462672 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" event={"ID":"f1f2d36b-4239-45fa-a6d7-85c316bd426e","Type":"ContainerStarted","Data":"b3b8930f46c8946861bc657ac9801445084228934bffa94247778652b2174d82"} Jul 06 19:28:23.468766 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:23.468727 2578 generic.go:356] "Generic (PLEG): container finished" podID="f1f2d36b-4239-45fa-a6d7-85c316bd426e" containerID="b3b8930f46c8946861bc657ac9801445084228934bffa94247778652b2174d82" exitCode=0 Jul 06 19:28:23.469215 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:23.468810 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" event={"ID":"f1f2d36b-4239-45fa-a6d7-85c316bd426e","Type":"ContainerDied","Data":"b3b8930f46c8946861bc657ac9801445084228934bffa94247778652b2174d82"} Jul 06 19:28:24.476422 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:24.476384 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" event={"ID":"f1f2d36b-4239-45fa-a6d7-85c316bd426e","Type":"ContainerStarted","Data":"8256b2550654e5c34e69b16380fdb1c7d2a644b9b56f133f6c13b9738c4f339b"} Jul 06 19:28:24.476871 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:24.476486 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:28:24.539095 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:24.539040 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" podStartSLOduration=2.341980811 podStartE2EDuration="6.539020622s" totalImagesPullingTime="3.118607922s" totalInitContainerRuntime="1.078431889s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:24.538503504 +0000 UTC m=+349.167457470" watchObservedRunningTime="2026-07-06 19:28:24.539020622 +0000 UTC m=+349.167974588" Jul 06 19:28:35.482734 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:35.482637 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:28:39.974948 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:39.974907 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kkjcw"] Jul 06 19:28:39.983561 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:39.983533 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:28:39.986553 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:39.986516 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 06 19:28:39.986712 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:39.986558 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-gzh7x\"" Jul 06 19:28:40.055250 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:40.055209 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kkjcw"] Jul 06 19:28:40.092102 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:40.092064 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxxsq\" (UniqueName: \"kubernetes.io/projected/85caf392-2962-4d96-8ae0-59dbf3366abe-kube-api-access-wxxsq\") pod \"odh-model-controller-7998c67dc4-kkjcw\" (UID: \"85caf392-2962-4d96-8ae0-59dbf3366abe\") " pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:28:40.092283 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:40.092108 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85caf392-2962-4d96-8ae0-59dbf3366abe-cert\") pod \"odh-model-controller-7998c67dc4-kkjcw\" (UID: \"85caf392-2962-4d96-8ae0-59dbf3366abe\") " pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:28:40.193331 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:40.193293 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxxsq\" (UniqueName: \"kubernetes.io/projected/85caf392-2962-4d96-8ae0-59dbf3366abe-kube-api-access-wxxsq\") pod \"odh-model-controller-7998c67dc4-kkjcw\" (UID: \"85caf392-2962-4d96-8ae0-59dbf3366abe\") " pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:28:40.193331 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:40.193339 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85caf392-2962-4d96-8ae0-59dbf3366abe-cert\") pod \"odh-model-controller-7998c67dc4-kkjcw\" (UID: \"85caf392-2962-4d96-8ae0-59dbf3366abe\") " pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:28:40.193627 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:28:40.193443 2578 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 06 19:28:40.193627 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:28:40.193502 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85caf392-2962-4d96-8ae0-59dbf3366abe-cert podName:85caf392-2962-4d96-8ae0-59dbf3366abe nodeName:}" failed. No retries permitted until 2026-07-06 19:28:40.693483764 +0000 UTC m=+365.322437712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/85caf392-2962-4d96-8ae0-59dbf3366abe-cert") pod "odh-model-controller-7998c67dc4-kkjcw" (UID: "85caf392-2962-4d96-8ae0-59dbf3366abe") : secret "odh-model-controller-webhook-cert" not found Jul 06 19:28:40.217058 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:40.217020 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxxsq\" (UniqueName: \"kubernetes.io/projected/85caf392-2962-4d96-8ae0-59dbf3366abe-kube-api-access-wxxsq\") pod \"odh-model-controller-7998c67dc4-kkjcw\" (UID: \"85caf392-2962-4d96-8ae0-59dbf3366abe\") " pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:28:40.697950 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:40.697904 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85caf392-2962-4d96-8ae0-59dbf3366abe-cert\") pod \"odh-model-controller-7998c67dc4-kkjcw\" (UID: \"85caf392-2962-4d96-8ae0-59dbf3366abe\") " pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:28:40.700354 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:40.700333 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85caf392-2962-4d96-8ae0-59dbf3366abe-cert\") pod \"odh-model-controller-7998c67dc4-kkjcw\" (UID: \"85caf392-2962-4d96-8ae0-59dbf3366abe\") " pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:28:40.894156 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:40.894120 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:28:41.060563 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:41.060529 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kkjcw"] Jul 06 19:28:41.064033 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:28:41.063997 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85caf392_2962_4d96_8ae0_59dbf3366abe.slice/crio-d159dab84d66fd1d5c61462d479eddd64e5c28e29c59fefe33a69438c6b513b2 WatchSource:0}: Error finding container d159dab84d66fd1d5c61462d479eddd64e5c28e29c59fefe33a69438c6b513b2: Status 404 returned error can't find the container with id d159dab84d66fd1d5c61462d479eddd64e5c28e29c59fefe33a69438c6b513b2 Jul 06 19:28:41.552841 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:41.552806 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" event={"ID":"85caf392-2962-4d96-8ae0-59dbf3366abe","Type":"ContainerStarted","Data":"d159dab84d66fd1d5c61462d479eddd64e5c28e29c59fefe33a69438c6b513b2"} Jul 06 19:28:44.574855 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:44.574819 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" event={"ID":"85caf392-2962-4d96-8ae0-59dbf3366abe","Type":"ContainerStarted","Data":"dae5d7722ca62b5791f0079e17a53145d49f58a6b1dc33d14959760a0abb608d"} Jul 06 19:28:44.575303 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:44.574959 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:28:44.594091 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:44.594039 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" podStartSLOduration=2.767052006 podStartE2EDuration="5.594018726s" totalImagesPullingTime="2.82696672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:44.592252335 +0000 UTC m=+369.221206302" watchObservedRunningTime="2026-07-06 19:28:44.594018726 +0000 UTC m=+369.222972739" Jul 06 19:28:50.360727 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:50.360651 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-67db589d55-p5z7q"] Jul 06 19:28:50.363102 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:50.363080 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67db589d55-p5z7q" Jul 06 19:28:50.365835 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:50.365804 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 19:28:50.368607 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:50.368590 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 19:28:50.389382 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:50.389351 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67db589d55-p5z7q"] Jul 06 19:28:50.480093 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:50.480054 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbws9\" (UniqueName: \"kubernetes.io/projected/2342b288-6681-49f1-85ca-261c3fe2e33d-kube-api-access-rbws9\") pod \"kube-auth-proxy-67db589d55-p5z7q\" (UID: \"2342b288-6681-49f1-85ca-261c3fe2e33d\") " pod="openshift-ingress/kube-auth-proxy-67db589d55-p5z7q" Jul 06 19:28:50.480285 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:50.480138 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2342b288-6681-49f1-85ca-261c3fe2e33d-tmp\") pod \"kube-auth-proxy-67db589d55-p5z7q\" (UID: \"2342b288-6681-49f1-85ca-261c3fe2e33d\") " pod="openshift-ingress/kube-auth-proxy-67db589d55-p5z7q" Jul 06 19:28:50.480285 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:50.480189 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2342b288-6681-49f1-85ca-261c3fe2e33d-tls-certs\") pod \"kube-auth-proxy-67db589d55-p5z7q\" (UID: \"2342b288-6681-49f1-85ca-261c3fe2e33d\") " pod="openshift-ingress/kube-auth-proxy-67db589d55-p5z7q" Jul 06 19:28:50.580985 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:50.580947 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2342b288-6681-49f1-85ca-261c3fe2e33d-tmp\") pod \"kube-auth-proxy-67db589d55-p5z7q\" (UID: \"2342b288-6681-49f1-85ca-261c3fe2e33d\") " pod="openshift-ingress/kube-auth-proxy-67db589d55-p5z7q" Jul 06 19:28:50.581178 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:50.581000 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2342b288-6681-49f1-85ca-261c3fe2e33d-tls-certs\") pod \"kube-auth-proxy-67db589d55-p5z7q\" (UID: \"2342b288-6681-49f1-85ca-261c3fe2e33d\") " pod="openshift-ingress/kube-auth-proxy-67db589d55-p5z7q" Jul 06 19:28:50.581178 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:50.581028 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbws9\" (UniqueName: \"kubernetes.io/projected/2342b288-6681-49f1-85ca-261c3fe2e33d-kube-api-access-rbws9\") pod \"kube-auth-proxy-67db589d55-p5z7q\" (UID: \"2342b288-6681-49f1-85ca-261c3fe2e33d\") " pod="openshift-ingress/kube-auth-proxy-67db589d55-p5z7q" Jul 06 19:28:50.583227 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:50.583200 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2342b288-6681-49f1-85ca-261c3fe2e33d-tmp\") pod \"kube-auth-proxy-67db589d55-p5z7q\" (UID: \"2342b288-6681-49f1-85ca-261c3fe2e33d\") " pod="openshift-ingress/kube-auth-proxy-67db589d55-p5z7q" Jul 06 19:28:50.583618 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:50.583595 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2342b288-6681-49f1-85ca-261c3fe2e33d-tls-certs\") pod \"kube-auth-proxy-67db589d55-p5z7q\" (UID: \"2342b288-6681-49f1-85ca-261c3fe2e33d\") " pod="openshift-ingress/kube-auth-proxy-67db589d55-p5z7q" Jul 06 19:28:50.592562 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:50.592531 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbws9\" (UniqueName: \"kubernetes.io/projected/2342b288-6681-49f1-85ca-261c3fe2e33d-kube-api-access-rbws9\") pod \"kube-auth-proxy-67db589d55-p5z7q\" (UID: \"2342b288-6681-49f1-85ca-261c3fe2e33d\") " pod="openshift-ingress/kube-auth-proxy-67db589d55-p5z7q" Jul 06 19:28:50.673821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:50.673724 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67db589d55-p5z7q" Jul 06 19:28:50.857917 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:50.857891 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67db589d55-p5z7q"] Jul 06 19:28:50.860308 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:28:50.860272 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2342b288_6681_49f1_85ca_261c3fe2e33d.slice/crio-2ffd13d5c4521a176d91afa981979c24cc6785acf2a2ddd2f2b01d998f4b5bc2 WatchSource:0}: Error finding container 2ffd13d5c4521a176d91afa981979c24cc6785acf2a2ddd2f2b01d998f4b5bc2: Status 404 returned error can't find the container with id 2ffd13d5c4521a176d91afa981979c24cc6785acf2a2ddd2f2b01d998f4b5bc2 Jul 06 19:28:51.607148 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:51.607109 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67db589d55-p5z7q" event={"ID":"2342b288-6681-49f1-85ca-261c3fe2e33d","Type":"ContainerStarted","Data":"2ffd13d5c4521a176d91afa981979c24cc6785acf2a2ddd2f2b01d998f4b5bc2"} Jul 06 19:28:52.547518 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.547482 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5"] Jul 06 19:28:52.549924 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.549905 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:28:52.590751 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.590549 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 19:28:52.590957 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.590934 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-86d8f\"" Jul 06 19:28:52.593637 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.593608 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5"] Jul 06 19:28:52.596382 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.596356 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5"] Jul 06 19:28:52.596527 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.596470 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:28:52.599121 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.599096 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 06 19:28:52.599467 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.599442 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-nfn7w\"" Jul 06 19:28:52.637905 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.637794 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5"] Jul 06 19:28:52.701992 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.701950 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vx2z\" (UniqueName: \"kubernetes.io/projected/40393be6-bd01-4617-ace4-5cbdfcd186fd-kube-api-access-6vx2z\") pod \"kserve-controller-manager-d6f4b65b8-pv5d5\" (UID: \"40393be6-bd01-4617-ace4-5cbdfcd186fd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:28:52.701992 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.701994 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40393be6-bd01-4617-ace4-5cbdfcd186fd-cert\") pod \"kserve-controller-manager-d6f4b65b8-pv5d5\" (UID: \"40393be6-bd01-4617-ace4-5cbdfcd186fd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:28:52.702266 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.702012 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f55184b1-bd4b-4da5-9f7a-d6099ed678ab-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hrvh5\" (UID: \"f55184b1-bd4b-4da5-9f7a-d6099ed678ab\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:28:52.702266 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.702100 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfpsl\" (UniqueName: \"kubernetes.io/projected/f55184b1-bd4b-4da5-9f7a-d6099ed678ab-kube-api-access-bfpsl\") pod \"llmisvc-controller-manager-777c6bd7d5-hrvh5\" (UID: \"f55184b1-bd4b-4da5-9f7a-d6099ed678ab\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:28:52.803094 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.802996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfpsl\" (UniqueName: \"kubernetes.io/projected/f55184b1-bd4b-4da5-9f7a-d6099ed678ab-kube-api-access-bfpsl\") pod \"llmisvc-controller-manager-777c6bd7d5-hrvh5\" (UID: \"f55184b1-bd4b-4da5-9f7a-d6099ed678ab\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:28:52.803094 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.803049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vx2z\" (UniqueName: \"kubernetes.io/projected/40393be6-bd01-4617-ace4-5cbdfcd186fd-kube-api-access-6vx2z\") pod \"kserve-controller-manager-d6f4b65b8-pv5d5\" (UID: \"40393be6-bd01-4617-ace4-5cbdfcd186fd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:28:52.803094 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.803077 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40393be6-bd01-4617-ace4-5cbdfcd186fd-cert\") pod \"kserve-controller-manager-d6f4b65b8-pv5d5\" (UID: \"40393be6-bd01-4617-ace4-5cbdfcd186fd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:28:52.803094 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.803095 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f55184b1-bd4b-4da5-9f7a-d6099ed678ab-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hrvh5\" (UID: \"f55184b1-bd4b-4da5-9f7a-d6099ed678ab\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:28:52.803360 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:28:52.803235 2578 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 06 19:28:52.803360 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:28:52.803301 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f55184b1-bd4b-4da5-9f7a-d6099ed678ab-cert podName:f55184b1-bd4b-4da5-9f7a-d6099ed678ab nodeName:}" failed. No retries permitted until 2026-07-06 19:28:53.303280537 +0000 UTC m=+377.932234485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f55184b1-bd4b-4da5-9f7a-d6099ed678ab-cert") pod "llmisvc-controller-manager-777c6bd7d5-hrvh5" (UID: "f55184b1-bd4b-4da5-9f7a-d6099ed678ab") : secret "llmisvc-webhook-server-cert" not found Jul 06 19:28:52.805583 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.805560 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40393be6-bd01-4617-ace4-5cbdfcd186fd-cert\") pod \"kserve-controller-manager-d6f4b65b8-pv5d5\" (UID: \"40393be6-bd01-4617-ace4-5cbdfcd186fd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:28:52.819881 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.819851 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfpsl\" (UniqueName: \"kubernetes.io/projected/f55184b1-bd4b-4da5-9f7a-d6099ed678ab-kube-api-access-bfpsl\") pod \"llmisvc-controller-manager-777c6bd7d5-hrvh5\" (UID: \"f55184b1-bd4b-4da5-9f7a-d6099ed678ab\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:28:52.826087 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.826058 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vx2z\" (UniqueName: \"kubernetes.io/projected/40393be6-bd01-4617-ace4-5cbdfcd186fd-kube-api-access-6vx2z\") pod \"kserve-controller-manager-d6f4b65b8-pv5d5\" (UID: \"40393be6-bd01-4617-ace4-5cbdfcd186fd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:28:52.863639 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:52.863607 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:28:53.060175 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:53.060142 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5"] Jul 06 19:28:53.309189 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:53.309098 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f55184b1-bd4b-4da5-9f7a-d6099ed678ab-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hrvh5\" (UID: \"f55184b1-bd4b-4da5-9f7a-d6099ed678ab\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:28:53.311973 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:53.311949 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f55184b1-bd4b-4da5-9f7a-d6099ed678ab-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hrvh5\" (UID: \"f55184b1-bd4b-4da5-9f7a-d6099ed678ab\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:28:53.517000 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:53.516962 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:28:53.610570 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:28:53.610500 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40393be6_bd01_4617_ace4_5cbdfcd186fd.slice/crio-1c27ac3840ba5dbf2b85c4ae35ae4a61158a1a3e2eb32f61bd2f3acb3e87bdc7 WatchSource:0}: Error finding container 1c27ac3840ba5dbf2b85c4ae35ae4a61158a1a3e2eb32f61bd2f3acb3e87bdc7: Status 404 returned error can't find the container with id 1c27ac3840ba5dbf2b85c4ae35ae4a61158a1a3e2eb32f61bd2f3acb3e87bdc7 Jul 06 19:28:53.625912 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:53.625883 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" event={"ID":"40393be6-bd01-4617-ace4-5cbdfcd186fd","Type":"ContainerStarted","Data":"1c27ac3840ba5dbf2b85c4ae35ae4a61158a1a3e2eb32f61bd2f3acb3e87bdc7"} Jul 06 19:28:53.789803 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:53.789767 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5"] Jul 06 19:28:53.798901 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:28:53.798872 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf55184b1_bd4b_4da5_9f7a_d6099ed678ab.slice/crio-4b2af9fbae4bb9cfaa7e811d2c6c34949325e8527c9c94f8fdc1d97812ffb3a0 WatchSource:0}: Error finding container 4b2af9fbae4bb9cfaa7e811d2c6c34949325e8527c9c94f8fdc1d97812ffb3a0: Status 404 returned error can't find the container with id 4b2af9fbae4bb9cfaa7e811d2c6c34949325e8527c9c94f8fdc1d97812ffb3a0 Jul 06 19:28:54.633490 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:54.633417 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67db589d55-p5z7q" event={"ID":"2342b288-6681-49f1-85ca-261c3fe2e33d","Type":"ContainerStarted","Data":"7d2ef69f207d3e1ceedcaae2481fbff02acf83445b7cdf1a6f0b4018bfab5c80"} Jul 06 19:28:54.634892 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:54.634841 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" event={"ID":"f55184b1-bd4b-4da5-9f7a-d6099ed678ab","Type":"ContainerStarted","Data":"4b2af9fbae4bb9cfaa7e811d2c6c34949325e8527c9c94f8fdc1d97812ffb3a0"} Jul 06 19:28:54.702003 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:54.701604 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-67db589d55-p5z7q" podStartSLOduration=1.9250170500000001 podStartE2EDuration="4.701581483s" totalImagesPullingTime="2.776564433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:54.700800526 +0000 UTC m=+379.329754507" watchObservedRunningTime="2026-07-06 19:28:54.701581483 +0000 UTC m=+379.330535446" Jul 06 19:28:55.582310 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:55.582280 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:28:56.654677 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:56.654596 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" event={"ID":"40393be6-bd01-4617-ace4-5cbdfcd186fd","Type":"ContainerStarted","Data":"e8adacd47c5619e2638d920f44a2af81ab817605a3e9faa1368c237893d034ce"} Jul 06 19:28:56.655159 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:56.654875 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:28:56.688054 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:56.687997 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" podStartSLOduration=1.825391539 podStartE2EDuration="4.687981312s" totalImagesPullingTime="2.862589773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:56.686454413 +0000 UTC m=+381.315408379" watchObservedRunningTime="2026-07-06 19:28:56.687981312 +0000 UTC m=+381.316935297" Jul 06 19:28:57.660299 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:57.660259 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" event={"ID":"f55184b1-bd4b-4da5-9f7a-d6099ed678ab","Type":"ContainerStarted","Data":"30ae76a8df34415c68920d893fb1a6d73aae5c5bb4ddc0fc5f600b867b394c19"} Jul 06 19:28:57.660724 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:57.660480 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:28:59.662358 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.662294 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" podStartSLOduration=4.3530040549999995 podStartE2EDuration="7.662270366s" totalImagesPullingTime="3.309266311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:57.920945896 +0000 UTC m=+382.549899862" watchObservedRunningTime="2026-07-06 19:28:59.662270366 +0000 UTC m=+384.291224332" Jul 06 19:28:59.663791 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.663771 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq"] Jul 06 19:28:59.667281 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.667259 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.669869 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.669840 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-xjh62\"" Jul 06 19:28:59.670099 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.670085 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 19:28:59.694309 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.694281 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq"] Jul 06 19:28:59.766980 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.766946 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/373da31a-f406-40cb-b233-e4e7f2cbf777-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.766980 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.766980 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/373da31a-f406-40cb-b233-e4e7f2cbf777-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.767239 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.767016 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/373da31a-f406-40cb-b233-e4e7f2cbf777-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.767239 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.767093 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/373da31a-f406-40cb-b233-e4e7f2cbf777-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.767239 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.767122 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/373da31a-f406-40cb-b233-e4e7f2cbf777-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.767239 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.767140 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/373da31a-f406-40cb-b233-e4e7f2cbf777-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.767239 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.767172 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/373da31a-f406-40cb-b233-e4e7f2cbf777-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.767239 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.767198 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6zdd\" (UniqueName: \"kubernetes.io/projected/373da31a-f406-40cb-b233-e4e7f2cbf777-kube-api-access-d6zdd\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.767239 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.767239 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/373da31a-f406-40cb-b233-e4e7f2cbf777-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.868454 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.868417 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/373da31a-f406-40cb-b233-e4e7f2cbf777-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.868649 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.868468 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/373da31a-f406-40cb-b233-e4e7f2cbf777-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.868649 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.868493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/373da31a-f406-40cb-b233-e4e7f2cbf777-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.868649 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.868514 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/373da31a-f406-40cb-b233-e4e7f2cbf777-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.868649 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.868536 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/373da31a-f406-40cb-b233-e4e7f2cbf777-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.868649 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.868554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zdd\" (UniqueName: \"kubernetes.io/projected/373da31a-f406-40cb-b233-e4e7f2cbf777-kube-api-access-d6zdd\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.868649 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.868580 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/373da31a-f406-40cb-b233-e4e7f2cbf777-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.868649 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.868612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/373da31a-f406-40cb-b233-e4e7f2cbf777-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.868649 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.868638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/373da31a-f406-40cb-b233-e4e7f2cbf777-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.869146 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.869051 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/373da31a-f406-40cb-b233-e4e7f2cbf777-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.869146 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.869102 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/373da31a-f406-40cb-b233-e4e7f2cbf777-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.869256 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.869161 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/373da31a-f406-40cb-b233-e4e7f2cbf777-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.869256 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.869236 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/373da31a-f406-40cb-b233-e4e7f2cbf777-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.869256 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.869246 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/373da31a-f406-40cb-b233-e4e7f2cbf777-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.870873 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.870845 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/373da31a-f406-40cb-b233-e4e7f2cbf777-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.871151 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.871134 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/373da31a-f406-40cb-b233-e4e7f2cbf777-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.888385 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.888344 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/373da31a-f406-40cb-b233-e4e7f2cbf777-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.888512 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.888430 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6zdd\" (UniqueName: \"kubernetes.io/projected/373da31a-f406-40cb-b233-e4e7f2cbf777-kube-api-access-d6zdd\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6786q4mq\" (UID: \"373da31a-f406-40cb-b233-e4e7f2cbf777\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:28:59.978903 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:28:59.978808 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:29:00.197143 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:00.197115 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq"] Jul 06 19:29:00.197255 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:29:00.197221 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373da31a_f406_40cb_b233_e4e7f2cbf777.slice/crio-53d4788cc792c98dd21d7cf0a648e2542b3823f7d2cb3181a4ba74f06ac98fcc WatchSource:0}: Error finding container 53d4788cc792c98dd21d7cf0a648e2542b3823f7d2cb3181a4ba74f06ac98fcc: Status 404 returned error can't find the container with id 53d4788cc792c98dd21d7cf0a648e2542b3823f7d2cb3181a4ba74f06ac98fcc Jul 06 19:29:00.676689 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:00.676650 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" event={"ID":"373da31a-f406-40cb-b233-e4e7f2cbf777","Type":"ContainerStarted","Data":"53d4788cc792c98dd21d7cf0a648e2542b3823f7d2cb3181a4ba74f06ac98fcc"} Jul 06 19:29:03.014310 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:03.014261 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:29:03.014708 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:03.014353 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:29:03.014708 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:03.014385 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:29:03.694014 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:03.693975 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" event={"ID":"373da31a-f406-40cb-b233-e4e7f2cbf777","Type":"ContainerStarted","Data":"53c628e90fa46b3b5706c4ebfaecf79dae7da934cade069529e92b34296fb347"} Jul 06 19:29:03.721476 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:03.721430 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" podStartSLOduration=1.9066534370000001 podStartE2EDuration="4.721416435s" totalImagesPullingTime="2.814762998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:29:03.718669291 +0000 UTC m=+388.347623257" watchObservedRunningTime="2026-07-06 19:29:03.721416435 +0000 UTC m=+388.350370402" Jul 06 19:29:03.979913 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:03.979827 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:29:04.984327 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:04.984293 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:29:05.702760 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:05.702729 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:29:05.703816 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:05.703797 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6786q4mq" Jul 06 19:29:27.666150 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:27.666119 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:29:28.667557 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:28.667527 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:29:39.949500 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:39.949461 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tllbh"] Jul 06 19:29:39.960647 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:39.960623 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tllbh" Jul 06 19:29:39.967951 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:39.967925 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:29:39.968822 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:39.968329 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:29:39.968822 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:39.968549 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-bmvfv\"" Jul 06 19:29:39.970539 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:39.970516 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tllbh"] Jul 06 19:29:40.107531 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:40.107497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gdwf\" (UniqueName: \"kubernetes.io/projected/1225f905-e2a4-4013-81d5-1520f8fe264e-kube-api-access-6gdwf\") pod \"kuadrant-operator-catalog-tllbh\" (UID: \"1225f905-e2a4-4013-81d5-1520f8fe264e\") " pod="kuadrant-system/kuadrant-operator-catalog-tllbh" Jul 06 19:29:40.167871 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:40.167838 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tllbh"] Jul 06 19:29:40.168104 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:29:40.168087 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6gdwf], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-catalog-tllbh" podUID="1225f905-e2a4-4013-81d5-1520f8fe264e" Jul 06 19:29:40.208651 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:40.208564 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gdwf\" (UniqueName: \"kubernetes.io/projected/1225f905-e2a4-4013-81d5-1520f8fe264e-kube-api-access-6gdwf\") pod \"kuadrant-operator-catalog-tllbh\" (UID: \"1225f905-e2a4-4013-81d5-1520f8fe264e\") " pod="kuadrant-system/kuadrant-operator-catalog-tllbh" Jul 06 19:29:40.237576 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:40.237541 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gdwf\" (UniqueName: \"kubernetes.io/projected/1225f905-e2a4-4013-81d5-1520f8fe264e-kube-api-access-6gdwf\") pod \"kuadrant-operator-catalog-tllbh\" (UID: \"1225f905-e2a4-4013-81d5-1520f8fe264e\") " pod="kuadrant-system/kuadrant-operator-catalog-tllbh" Jul 06 19:29:40.379710 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:40.379663 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-84jdz"] Jul 06 19:29:40.383133 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:40.383116 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-84jdz" Jul 06 19:29:40.395523 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:40.395498 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-84jdz"] Jul 06 19:29:40.512232 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:40.512149 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfdgt\" (UniqueName: \"kubernetes.io/projected/5bd6e3a8-f67a-4255-b7c9-13958774c94c-kube-api-access-lfdgt\") pod \"kuadrant-operator-catalog-84jdz\" (UID: \"5bd6e3a8-f67a-4255-b7c9-13958774c94c\") " pod="kuadrant-system/kuadrant-operator-catalog-84jdz" Jul 06 19:29:40.612552 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:40.612511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfdgt\" (UniqueName: \"kubernetes.io/projected/5bd6e3a8-f67a-4255-b7c9-13958774c94c-kube-api-access-lfdgt\") pod \"kuadrant-operator-catalog-84jdz\" (UID: \"5bd6e3a8-f67a-4255-b7c9-13958774c94c\") " pod="kuadrant-system/kuadrant-operator-catalog-84jdz" Jul 06 19:29:40.623562 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:40.623524 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfdgt\" (UniqueName: \"kubernetes.io/projected/5bd6e3a8-f67a-4255-b7c9-13958774c94c-kube-api-access-lfdgt\") pod \"kuadrant-operator-catalog-84jdz\" (UID: \"5bd6e3a8-f67a-4255-b7c9-13958774c94c\") " pod="kuadrant-system/kuadrant-operator-catalog-84jdz" Jul 06 19:29:40.692679 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:40.692643 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-84jdz" Jul 06 19:29:40.826544 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:40.826391 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-84jdz"] Jul 06 19:29:40.829376 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:29:40.829351 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd6e3a8_f67a_4255_b7c9_13958774c94c.slice/crio-bc79f36e5e59e4f9e888137cd6b439806b1e06cba8e6c92198fb0f0cb3198a8e WatchSource:0}: Error finding container bc79f36e5e59e4f9e888137cd6b439806b1e06cba8e6c92198fb0f0cb3198a8e: Status 404 returned error can't find the container with id bc79f36e5e59e4f9e888137cd6b439806b1e06cba8e6c92198fb0f0cb3198a8e Jul 06 19:29:40.866414 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:40.866382 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-84jdz" event={"ID":"5bd6e3a8-f67a-4255-b7c9-13958774c94c","Type":"ContainerStarted","Data":"bc79f36e5e59e4f9e888137cd6b439806b1e06cba8e6c92198fb0f0cb3198a8e"} Jul 06 19:29:40.866414 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:40.866408 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tllbh" Jul 06 19:29:40.874827 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:40.874803 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tllbh" Jul 06 19:29:41.015875 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:41.015835 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gdwf\" (UniqueName: \"kubernetes.io/projected/1225f905-e2a4-4013-81d5-1520f8fe264e-kube-api-access-6gdwf\") pod \"1225f905-e2a4-4013-81d5-1520f8fe264e\" (UID: \"1225f905-e2a4-4013-81d5-1520f8fe264e\") " Jul 06 19:29:41.017928 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:41.017907 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1225f905-e2a4-4013-81d5-1520f8fe264e-kube-api-access-6gdwf" (OuterVolumeSpecName: "kube-api-access-6gdwf") pod "1225f905-e2a4-4013-81d5-1520f8fe264e" (UID: "1225f905-e2a4-4013-81d5-1520f8fe264e"). InnerVolumeSpecName "kube-api-access-6gdwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:29:41.116828 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:41.116736 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gdwf\" (UniqueName: \"kubernetes.io/projected/1225f905-e2a4-4013-81d5-1520f8fe264e-kube-api-access-6gdwf\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:29:41.871492 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:41.871462 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tllbh" Jul 06 19:29:41.917904 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:41.917871 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tllbh"] Jul 06 19:29:41.928889 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:41.928857 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tllbh"] Jul 06 19:29:43.874895 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:43.874862 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1225f905-e2a4-4013-81d5-1520f8fe264e" path="/var/lib/kubelet/pods/1225f905-e2a4-4013-81d5-1520f8fe264e/volumes" Jul 06 19:29:43.881358 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:43.881325 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-84jdz" event={"ID":"5bd6e3a8-f67a-4255-b7c9-13958774c94c","Type":"ContainerStarted","Data":"ea56b2940521980a302fbe570e1714794825f18e32722074cb0f596f0e4549a1"} Jul 06 19:29:43.900455 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:43.900409 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-84jdz" podStartSLOduration=1.595282712 podStartE2EDuration="3.900393334s" totalImagesPullingTime="2.305110622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:29:43.898930734 +0000 UTC m=+428.527884700" watchObservedRunningTime="2026-07-06 19:29:43.900393334 +0000 UTC m=+428.529347308" Jul 06 19:29:50.693653 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:50.693613 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-84jdz" Jul 06 19:29:50.694123 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:50.693684 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-84jdz" Jul 06 19:29:50.717012 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:50.716983 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-84jdz" Jul 06 19:29:50.914061 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:50.914028 2578 generic.go:356] "Generic (PLEG): container finished" podID="85caf392-2962-4d96-8ae0-59dbf3366abe" containerID="dae5d7722ca62b5791f0079e17a53145d49f58a6b1dc33d14959760a0abb608d" exitCode=1 Jul 06 19:29:50.914236 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:50.914100 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" event={"ID":"85caf392-2962-4d96-8ae0-59dbf3366abe","Type":"ContainerDied","Data":"dae5d7722ca62b5791f0079e17a53145d49f58a6b1dc33d14959760a0abb608d"} Jul 06 19:29:50.914492 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:50.914471 2578 scope.go:117] "RemoveContainer" containerID="dae5d7722ca62b5791f0079e17a53145d49f58a6b1dc33d14959760a0abb608d" Jul 06 19:29:50.937491 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:50.937469 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-84jdz" Jul 06 19:29:51.922919 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:51.922885 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" event={"ID":"85caf392-2962-4d96-8ae0-59dbf3366abe","Type":"ContainerStarted","Data":"6cc1fcfc63a5cb0661d419c23e02e8c07c2efa578f29b555f8b367336c2fc2af"} Jul 06 19:29:51.923330 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:29:51.923236 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:30:02.930043 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:02.929972 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:30:28.866185 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:28.866148 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz"] Jul 06 19:30:28.868298 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:28.868283 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" Jul 06 19:30:28.870202 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:28.870181 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-sfbsq\"" Jul 06 19:30:28.886005 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:28.885981 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz"] Jul 06 19:30:28.910659 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:28.910625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8mwb\" (UniqueName: \"kubernetes.io/projected/7f55d1c1-4236-45e8-812a-68f2b9015ff6-kube-api-access-g8mwb\") pod \"authorino-operator-75cc8d5cb7-ndlnz\" (UID: \"7f55d1c1-4236-45e8-812a-68f2b9015ff6\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" Jul 06 19:30:29.011351 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:29.011310 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8mwb\" (UniqueName: \"kubernetes.io/projected/7f55d1c1-4236-45e8-812a-68f2b9015ff6-kube-api-access-g8mwb\") pod \"authorino-operator-75cc8d5cb7-ndlnz\" (UID: \"7f55d1c1-4236-45e8-812a-68f2b9015ff6\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" Jul 06 19:30:29.026775 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:29.026746 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8mwb\" (UniqueName: \"kubernetes.io/projected/7f55d1c1-4236-45e8-812a-68f2b9015ff6-kube-api-access-g8mwb\") pod \"authorino-operator-75cc8d5cb7-ndlnz\" (UID: \"7f55d1c1-4236-45e8-812a-68f2b9015ff6\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" Jul 06 19:30:29.177825 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:29.177728 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" Jul 06 19:30:29.322711 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:29.322671 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz"] Jul 06 19:30:29.323958 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:30:29.323931 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f55d1c1_4236_45e8_812a_68f2b9015ff6.slice/crio-c806fe755a1a1ad08869f51544a39d986f544f9930600e5955fce4a8bd6c2b80 WatchSource:0}: Error finding container c806fe755a1a1ad08869f51544a39d986f544f9930600e5955fce4a8bd6c2b80: Status 404 returned error can't find the container with id c806fe755a1a1ad08869f51544a39d986f544f9930600e5955fce4a8bd6c2b80 Jul 06 19:30:30.101911 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:30.101834 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" event={"ID":"7f55d1c1-4236-45e8-812a-68f2b9015ff6","Type":"ContainerStarted","Data":"c806fe755a1a1ad08869f51544a39d986f544f9930600e5955fce4a8bd6c2b80"} Jul 06 19:30:31.485975 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:31.485948 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4"] Jul 06 19:30:31.488946 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:31.488926 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4" Jul 06 19:30:31.492026 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:31.492009 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dwzhg\"" Jul 06 19:30:31.511574 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:31.511545 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4"] Jul 06 19:30:31.533807 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:31.533785 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8cnj\" (UniqueName: \"kubernetes.io/projected/c7fa35c6-9cb8-4423-9245-e0cde0c2b353-kube-api-access-z8cnj\") pod \"limitador-operator-controller-manager-859d7b78bd-k87s4\" (UID: \"c7fa35c6-9cb8-4423-9245-e0cde0c2b353\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4" Jul 06 19:30:31.634949 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:31.634908 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8cnj\" (UniqueName: \"kubernetes.io/projected/c7fa35c6-9cb8-4423-9245-e0cde0c2b353-kube-api-access-z8cnj\") pod \"limitador-operator-controller-manager-859d7b78bd-k87s4\" (UID: \"c7fa35c6-9cb8-4423-9245-e0cde0c2b353\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4" Jul 06 19:30:31.650258 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:31.650224 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8cnj\" (UniqueName: \"kubernetes.io/projected/c7fa35c6-9cb8-4423-9245-e0cde0c2b353-kube-api-access-z8cnj\") pod \"limitador-operator-controller-manager-859d7b78bd-k87s4\" (UID: \"c7fa35c6-9cb8-4423-9245-e0cde0c2b353\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4" Jul 06 19:30:31.815017 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:31.814986 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4" Jul 06 19:30:31.968637 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:31.968607 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4"] Jul 06 19:30:31.972589 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:30:31.972559 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fa35c6_9cb8_4423_9245_e0cde0c2b353.slice/crio-8aaf37d0f7b8848232c7e352bb2cf1a12a5d0a7c186fa8402c3cfd322d58fed8 WatchSource:0}: Error finding container 8aaf37d0f7b8848232c7e352bb2cf1a12a5d0a7c186fa8402c3cfd322d58fed8: Status 404 returned error can't find the container with id 8aaf37d0f7b8848232c7e352bb2cf1a12a5d0a7c186fa8402c3cfd322d58fed8 Jul 06 19:30:32.112517 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:32.112427 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4" event={"ID":"c7fa35c6-9cb8-4423-9245-e0cde0c2b353","Type":"ContainerStarted","Data":"8aaf37d0f7b8848232c7e352bb2cf1a12a5d0a7c186fa8402c3cfd322d58fed8"} Jul 06 19:30:32.113987 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:32.113965 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" event={"ID":"7f55d1c1-4236-45e8-812a-68f2b9015ff6","Type":"ContainerStarted","Data":"fcbe2dbbb877039acd663056df24753798471b9700ebbed0eba515b535f8d27a"} Jul 06 19:30:32.114134 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:32.114077 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" Jul 06 19:30:32.140504 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:32.140453 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" podStartSLOduration=2.007471671 podStartE2EDuration="4.140436868s" totalImagesPullingTime="2.132965197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:30:32.139135577 +0000 UTC m=+476.768089543" watchObservedRunningTime="2026-07-06 19:30:32.140436868 +0000 UTC m=+476.769390886" Jul 06 19:30:34.126274 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:34.126230 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4" event={"ID":"c7fa35c6-9cb8-4423-9245-e0cde0c2b353","Type":"ContainerStarted","Data":"6351aa69ab4a8e2d54b3e0eec62ca8d7cb74d8563091b88bb3ee4c3531174bd5"} Jul 06 19:30:34.126742 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:34.126388 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4" Jul 06 19:30:34.145739 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:34.145636 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4" podStartSLOduration=1.443203394 podStartE2EDuration="3.145616985s" totalImagesPullingTime="1.702413591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:30:34.144813532 +0000 UTC m=+478.773767495" watchObservedRunningTime="2026-07-06 19:30:34.145616985 +0000 UTC m=+478.774570951" Jul 06 19:30:37.560510 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:37.560476 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w"] Jul 06 19:30:37.562950 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:37.562933 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" Jul 06 19:30:37.565920 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:37.565892 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-n9nr2\"" Jul 06 19:30:37.566060 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:37.565961 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 06 19:30:37.591393 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:37.591362 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w"] Jul 06 19:30:37.684217 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:37.684175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znltk\" (UniqueName: \"kubernetes.io/projected/18778df4-9ca2-41fe-a983-e69f2d564dd2-kube-api-access-znltk\") pod \"dns-operator-controller-manager-864dd78c4c-6br2w\" (UID: \"18778df4-9ca2-41fe-a983-e69f2d564dd2\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" Jul 06 19:30:37.784749 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:37.784674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znltk\" (UniqueName: \"kubernetes.io/projected/18778df4-9ca2-41fe-a983-e69f2d564dd2-kube-api-access-znltk\") pod \"dns-operator-controller-manager-864dd78c4c-6br2w\" (UID: \"18778df4-9ca2-41fe-a983-e69f2d564dd2\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" Jul 06 19:30:37.796513 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:37.796484 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znltk\" (UniqueName: \"kubernetes.io/projected/18778df4-9ca2-41fe-a983-e69f2d564dd2-kube-api-access-znltk\") pod \"dns-operator-controller-manager-864dd78c4c-6br2w\" (UID: \"18778df4-9ca2-41fe-a983-e69f2d564dd2\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" Jul 06 19:30:37.873153 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:37.873050 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" Jul 06 19:30:38.015171 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:38.015143 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w"] Jul 06 19:30:38.016825 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:30:38.016794 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18778df4_9ca2_41fe_a983_e69f2d564dd2.slice/crio-9f324e2b9eabac1c693a49357c199a21e7cc2919e8776c11316575fd08998fd1 WatchSource:0}: Error finding container 9f324e2b9eabac1c693a49357c199a21e7cc2919e8776c11316575fd08998fd1: Status 404 returned error can't find the container with id 9f324e2b9eabac1c693a49357c199a21e7cc2919e8776c11316575fd08998fd1 Jul 06 19:30:38.145273 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:38.145180 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" event={"ID":"18778df4-9ca2-41fe-a983-e69f2d564dd2","Type":"ContainerStarted","Data":"9f324e2b9eabac1c693a49357c199a21e7cc2919e8776c11316575fd08998fd1"} Jul 06 19:30:41.162316 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:41.162281 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" event={"ID":"18778df4-9ca2-41fe-a983-e69f2d564dd2","Type":"ContainerStarted","Data":"67fb9e8903cc9017727d9e8717a3ff14a41133cdda1c9288588ffcdc83acb962"} Jul 06 19:30:41.162730 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:41.162351 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" Jul 06 19:30:41.181427 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:41.181378 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" podStartSLOduration=1.986445765 podStartE2EDuration="4.181362572s" totalImagesPullingTime="2.194916807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:30:41.179482493 +0000 UTC m=+485.808436460" watchObservedRunningTime="2026-07-06 19:30:41.181362572 +0000 UTC m=+485.810316538" Jul 06 19:30:43.121918 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:43.121890 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" Jul 06 19:30:44.990158 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:44.990106 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lrmt9"] Jul 06 19:30:44.996111 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:44.996085 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lrmt9" Jul 06 19:30:44.999055 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:44.999034 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 06 19:30:44.999741 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:44.999722 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 06 19:30:45.000617 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:45.000603 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mg4xx\"" Jul 06 19:30:45.017581 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:45.017552 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lrmt9"] Jul 06 19:30:45.133493 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:45.133466 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4" Jul 06 19:30:45.154164 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:45.154134 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d343a282-5e48-4a26-acb0-e1818285d7eb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lrmt9\" (UID: \"d343a282-5e48-4a26-acb0-e1818285d7eb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lrmt9" Jul 06 19:30:45.154300 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:45.154176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbbjz\" (UniqueName: \"kubernetes.io/projected/d343a282-5e48-4a26-acb0-e1818285d7eb-kube-api-access-xbbjz\") pod \"kuadrant-console-plugin-66b5774985-lrmt9\" (UID: \"d343a282-5e48-4a26-acb0-e1818285d7eb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lrmt9" Jul 06 19:30:45.154300 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:45.154252 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d343a282-5e48-4a26-acb0-e1818285d7eb-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lrmt9\" (UID: \"d343a282-5e48-4a26-acb0-e1818285d7eb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lrmt9" Jul 06 19:30:45.255435 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:45.255336 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbbjz\" (UniqueName: \"kubernetes.io/projected/d343a282-5e48-4a26-acb0-e1818285d7eb-kube-api-access-xbbjz\") pod \"kuadrant-console-plugin-66b5774985-lrmt9\" (UID: \"d343a282-5e48-4a26-acb0-e1818285d7eb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lrmt9" Jul 06 19:30:45.255435 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:45.255387 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d343a282-5e48-4a26-acb0-e1818285d7eb-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lrmt9\" (UID: \"d343a282-5e48-4a26-acb0-e1818285d7eb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lrmt9" Jul 06 19:30:45.255659 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:45.255544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d343a282-5e48-4a26-acb0-e1818285d7eb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lrmt9\" (UID: \"d343a282-5e48-4a26-acb0-e1818285d7eb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lrmt9" Jul 06 19:30:45.256209 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:45.256184 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d343a282-5e48-4a26-acb0-e1818285d7eb-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lrmt9\" (UID: \"d343a282-5e48-4a26-acb0-e1818285d7eb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lrmt9" Jul 06 19:30:45.258082 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:45.258061 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d343a282-5e48-4a26-acb0-e1818285d7eb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lrmt9\" (UID: \"d343a282-5e48-4a26-acb0-e1818285d7eb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lrmt9" Jul 06 19:30:45.274310 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:45.274285 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbbjz\" (UniqueName: \"kubernetes.io/projected/d343a282-5e48-4a26-acb0-e1818285d7eb-kube-api-access-xbbjz\") pod \"kuadrant-console-plugin-66b5774985-lrmt9\" (UID: \"d343a282-5e48-4a26-acb0-e1818285d7eb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lrmt9" Jul 06 19:30:45.307904 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:45.307867 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lrmt9" Jul 06 19:30:45.453243 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:45.453223 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lrmt9"] Jul 06 19:30:45.456107 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:30:45.456079 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd343a282_5e48_4a26_acb0_e1818285d7eb.slice/crio-544f8dce42d72f8b8a74e31fbdfef2bb5d067e51419de14d396291bd3cec330a WatchSource:0}: Error finding container 544f8dce42d72f8b8a74e31fbdfef2bb5d067e51419de14d396291bd3cec330a: Status 404 returned error can't find the container with id 544f8dce42d72f8b8a74e31fbdfef2bb5d067e51419de14d396291bd3cec330a Jul 06 19:30:46.193118 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:46.193081 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lrmt9" event={"ID":"d343a282-5e48-4a26-acb0-e1818285d7eb","Type":"ContainerStarted","Data":"544f8dce42d72f8b8a74e31fbdfef2bb5d067e51419de14d396291bd3cec330a"} Jul 06 19:30:52.169790 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:52.169754 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" Jul 06 19:30:56.507884 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:56.507847 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4"] Jul 06 19:30:56.508408 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:56.508078 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4" podUID="c7fa35c6-9cb8-4423-9245-e0cde0c2b353" containerName="manager" containerID="cri-o://6351aa69ab4a8e2d54b3e0eec62ca8d7cb74d8563091b88bb3ee4c3531174bd5" gracePeriod=2 Jul 06 19:30:56.536960 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:56.536925 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4"] Jul 06 19:30:56.554338 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:30:56.554299 2578 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"c7fa35c6-9cb8-4423-9245-e0cde0c2b353\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-k87s4\": pods \"limitador-operator-controller-manager-859d7b78bd-k87s4\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4" Jul 06 19:30:56.558842 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:56.558813 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz"] Jul 06 19:30:56.559253 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:56.559237 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7fa35c6-9cb8-4423-9245-e0cde0c2b353" containerName="manager" Jul 06 19:30:56.559359 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:56.559256 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7fa35c6-9cb8-4423-9245-e0cde0c2b353" containerName="manager" Jul 06 19:30:56.559414 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:56.559366 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7fa35c6-9cb8-4423-9245-e0cde0c2b353" containerName="manager" Jul 06 19:30:56.564883 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:56.564860 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" Jul 06 19:30:56.568591 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:30:56.568557 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-k87s4\" is forbidden: User \"system:node:ip-10-0-140-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-94.ec2.internal' and this object" podUID="c7fa35c6-9cb8-4423-9245-e0cde0c2b353" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4" Jul 06 19:30:56.579352 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:56.579279 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz"] Jul 06 19:30:56.672458 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:56.672421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfsdw\" (UniqueName: \"kubernetes.io/projected/fc0a6e65-42be-43eb-8a55-363e166ecc16-kube-api-access-sfsdw\") pod \"limitador-operator-controller-manager-859d7b78bd-4lrbz\" (UID: \"fc0a6e65-42be-43eb-8a55-363e166ecc16\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" Jul 06 19:30:56.773714 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:56.773665 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfsdw\" (UniqueName: \"kubernetes.io/projected/fc0a6e65-42be-43eb-8a55-363e166ecc16-kube-api-access-sfsdw\") pod \"limitador-operator-controller-manager-859d7b78bd-4lrbz\" (UID: \"fc0a6e65-42be-43eb-8a55-363e166ecc16\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" Jul 06 19:30:56.788012 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:56.787982 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfsdw\" (UniqueName: \"kubernetes.io/projected/fc0a6e65-42be-43eb-8a55-363e166ecc16-kube-api-access-sfsdw\") pod \"limitador-operator-controller-manager-859d7b78bd-4lrbz\" (UID: \"fc0a6e65-42be-43eb-8a55-363e166ecc16\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" Jul 06 19:30:56.897828 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:30:56.897789 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" Jul 06 19:31:08.581575 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:08.581552 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz"] Jul 06 19:31:08.758207 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:31:08.758164 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0a6e65_42be_43eb_8a55_363e166ecc16.slice/crio-6f7efd193e8d02e0569551d9ec23fcd7501bc362f1394ed999ba1a068fb4e483 WatchSource:0}: Error finding container 6f7efd193e8d02e0569551d9ec23fcd7501bc362f1394ed999ba1a068fb4e483: Status 404 returned error can't find the container with id 6f7efd193e8d02e0569551d9ec23fcd7501bc362f1394ed999ba1a068fb4e483 Jul 06 19:31:08.791890 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:08.791865 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4" Jul 06 19:31:08.793896 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:31:08.793866 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-k87s4\" is forbidden: User \"system:node:ip-10-0-140-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-94.ec2.internal' and this object" podUID="c7fa35c6-9cb8-4423-9245-e0cde0c2b353" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4" Jul 06 19:31:08.887056 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:08.887032 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8cnj\" (UniqueName: \"kubernetes.io/projected/c7fa35c6-9cb8-4423-9245-e0cde0c2b353-kube-api-access-z8cnj\") pod \"c7fa35c6-9cb8-4423-9245-e0cde0c2b353\" (UID: \"c7fa35c6-9cb8-4423-9245-e0cde0c2b353\") " Jul 06 19:31:08.889367 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:08.889337 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fa35c6-9cb8-4423-9245-e0cde0c2b353-kube-api-access-z8cnj" (OuterVolumeSpecName: "kube-api-access-z8cnj") pod "c7fa35c6-9cb8-4423-9245-e0cde0c2b353" (UID: "c7fa35c6-9cb8-4423-9245-e0cde0c2b353"). InnerVolumeSpecName "kube-api-access-z8cnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:31:08.988081 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:08.988047 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8cnj\" (UniqueName: \"kubernetes.io/projected/c7fa35c6-9cb8-4423-9245-e0cde0c2b353-kube-api-access-z8cnj\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:31:09.334440 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:09.334406 2578 generic.go:356] "Generic (PLEG): container finished" podID="c7fa35c6-9cb8-4423-9245-e0cde0c2b353" containerID="6351aa69ab4a8e2d54b3e0eec62ca8d7cb74d8563091b88bb3ee4c3531174bd5" exitCode=0 Jul 06 19:31:09.334628 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:09.334452 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4" Jul 06 19:31:09.334628 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:09.334510 2578 scope.go:117] "RemoveContainer" containerID="6351aa69ab4a8e2d54b3e0eec62ca8d7cb74d8563091b88bb3ee4c3531174bd5" Jul 06 19:31:09.336122 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:09.336092 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lrmt9" event={"ID":"d343a282-5e48-4a26-acb0-e1818285d7eb","Type":"ContainerStarted","Data":"2557a408c0467684e873ff2966fcc0735e59faaf42bb330e97d54f91c5e7c533"} Jul 06 19:31:09.336564 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:31:09.336535 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-k87s4\" is forbidden: User \"system:node:ip-10-0-140-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-94.ec2.internal' and this object" podUID="c7fa35c6-9cb8-4423-9245-e0cde0c2b353" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4" Jul 06 19:31:09.337821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:09.337796 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" event={"ID":"fc0a6e65-42be-43eb-8a55-363e166ecc16","Type":"ContainerStarted","Data":"a621f2ae25ab18ce1e39c9bdf6a04cc658bada680dfa5e5f8dffbdcc84b924a4"} Jul 06 19:31:09.337931 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:09.337842 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" event={"ID":"fc0a6e65-42be-43eb-8a55-363e166ecc16","Type":"ContainerStarted","Data":"6f7efd193e8d02e0569551d9ec23fcd7501bc362f1394ed999ba1a068fb4e483"} Jul 06 19:31:09.337931 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:09.337891 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" Jul 06 19:31:09.338263 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:31:09.338242 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-k87s4\" is forbidden: User \"system:node:ip-10-0-140-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-94.ec2.internal' and this object" podUID="c7fa35c6-9cb8-4423-9245-e0cde0c2b353" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4" Jul 06 19:31:09.344372 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:09.344352 2578 scope.go:117] "RemoveContainer" containerID="6351aa69ab4a8e2d54b3e0eec62ca8d7cb74d8563091b88bb3ee4c3531174bd5" Jul 06 19:31:09.344626 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:31:09.344607 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6351aa69ab4a8e2d54b3e0eec62ca8d7cb74d8563091b88bb3ee4c3531174bd5\": container with ID starting with 6351aa69ab4a8e2d54b3e0eec62ca8d7cb74d8563091b88bb3ee4c3531174bd5 not found: ID does not exist" containerID="6351aa69ab4a8e2d54b3e0eec62ca8d7cb74d8563091b88bb3ee4c3531174bd5" Jul 06 19:31:09.344687 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:09.344640 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6351aa69ab4a8e2d54b3e0eec62ca8d7cb74d8563091b88bb3ee4c3531174bd5"} err="failed to get container status \"6351aa69ab4a8e2d54b3e0eec62ca8d7cb74d8563091b88bb3ee4c3531174bd5\": rpc error: code = NotFound desc = could not find container \"6351aa69ab4a8e2d54b3e0eec62ca8d7cb74d8563091b88bb3ee4c3531174bd5\": container with ID starting with 6351aa69ab4a8e2d54b3e0eec62ca8d7cb74d8563091b88bb3ee4c3531174bd5 not found: ID does not exist" Jul 06 19:31:09.377689 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:09.377631 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lrmt9" podStartSLOduration=2.016368964 podStartE2EDuration="25.377614694s" totalImagesPullingTime="23.36124573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:31:09.376398891 +0000 UTC m=+514.005352917" watchObservedRunningTime="2026-07-06 19:31:09.377614694 +0000 UTC m=+514.006568697" Jul 06 19:31:09.414609 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:09.414560 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" podStartSLOduration=13.414541538 podStartE2EDuration="13.414541538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:31:09.41431435 +0000 UTC m=+514.043268317" watchObservedRunningTime="2026-07-06 19:31:09.414541538 +0000 UTC m=+514.043495504" Jul 06 19:31:09.416498 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:31:09.416461 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-k87s4\" is forbidden: User \"system:node:ip-10-0-140-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-94.ec2.internal' and this object" podUID="c7fa35c6-9cb8-4423-9245-e0cde0c2b353" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k87s4" Jul 06 19:31:09.875281 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:09.875252 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7fa35c6-9cb8-4423-9245-e0cde0c2b353" path="/var/lib/kubelet/pods/c7fa35c6-9cb8-4423-9245-e0cde0c2b353/volumes" Jul 06 19:31:20.347025 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:20.346989 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" Jul 06 19:31:31.302051 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.301971 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p"] Jul 06 19:31:31.305316 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.305296 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.307591 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.307565 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-6gz9l\"" Jul 06 19:31:31.337510 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.337468 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p"] Jul 06 19:31:31.386007 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.385971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639168ba-36ec-47a3-a023-6135f27dd8d9-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.386205 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.386023 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639168ba-36ec-47a3-a023-6135f27dd8d9-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.386205 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.386051 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cgkc\" (UniqueName: \"kubernetes.io/projected/639168ba-36ec-47a3-a023-6135f27dd8d9-kube-api-access-4cgkc\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.386205 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.386130 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639168ba-36ec-47a3-a023-6135f27dd8d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.386205 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.386159 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639168ba-36ec-47a3-a023-6135f27dd8d9-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.386205 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.386177 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639168ba-36ec-47a3-a023-6135f27dd8d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.386412 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.386257 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639168ba-36ec-47a3-a023-6135f27dd8d9-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.386412 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.386310 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639168ba-36ec-47a3-a023-6135f27dd8d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.386412 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.386332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639168ba-36ec-47a3-a023-6135f27dd8d9-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.486825 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.486788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639168ba-36ec-47a3-a023-6135f27dd8d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.487029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.486828 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639168ba-36ec-47a3-a023-6135f27dd8d9-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.487029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.486864 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639168ba-36ec-47a3-a023-6135f27dd8d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.487029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.486903 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639168ba-36ec-47a3-a023-6135f27dd8d9-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.487029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.486933 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639168ba-36ec-47a3-a023-6135f27dd8d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.487029 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.486958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639168ba-36ec-47a3-a023-6135f27dd8d9-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.487305 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.487098 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639168ba-36ec-47a3-a023-6135f27dd8d9-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.487305 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.487180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639168ba-36ec-47a3-a023-6135f27dd8d9-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.487305 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.487207 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cgkc\" (UniqueName: \"kubernetes.io/projected/639168ba-36ec-47a3-a023-6135f27dd8d9-kube-api-access-4cgkc\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.487465 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.487389 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639168ba-36ec-47a3-a023-6135f27dd8d9-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.487465 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.487403 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639168ba-36ec-47a3-a023-6135f27dd8d9-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.487599 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.487556 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639168ba-36ec-47a3-a023-6135f27dd8d9-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.487683 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.487665 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639168ba-36ec-47a3-a023-6135f27dd8d9-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.488342 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.488315 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639168ba-36ec-47a3-a023-6135f27dd8d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.489484 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.489449 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639168ba-36ec-47a3-a023-6135f27dd8d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.489484 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.489467 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639168ba-36ec-47a3-a023-6135f27dd8d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.495416 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.495390 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639168ba-36ec-47a3-a023-6135f27dd8d9-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.495511 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.495478 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cgkc\" (UniqueName: \"kubernetes.io/projected/639168ba-36ec-47a3-a023-6135f27dd8d9-kube-api-access-4cgkc\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nxb4p\" (UID: \"639168ba-36ec-47a3-a023-6135f27dd8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.615737 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.615614 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:31.778296 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.778258 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p"] Jul 06 19:31:31.783178 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:31:31.783145 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639168ba_36ec_47a3_a023_6135f27dd8d9.slice/crio-ee0c2adf35acb9ce4e792147fb9669294a8e1316a51b93f00ba75e6aeffb448c WatchSource:0}: Error finding container ee0c2adf35acb9ce4e792147fb9669294a8e1316a51b93f00ba75e6aeffb448c: Status 404 returned error can't find the container with id ee0c2adf35acb9ce4e792147fb9669294a8e1316a51b93f00ba75e6aeffb448c Jul 06 19:31:31.785570 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.785527 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:31:31.785645 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.785628 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:31:31.785711 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:31.785677 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:31:32.453942 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:32.453904 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" event={"ID":"639168ba-36ec-47a3-a023-6135f27dd8d9","Type":"ContainerStarted","Data":"4489cb6ea53d92073eff97ea249973f56cb06cbdfe146e666943323bfc368209"} Jul 06 19:31:32.454354 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:32.453959 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" event={"ID":"639168ba-36ec-47a3-a023-6135f27dd8d9","Type":"ContainerStarted","Data":"ee0c2adf35acb9ce4e792147fb9669294a8e1316a51b93f00ba75e6aeffb448c"} Jul 06 19:31:32.478490 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:32.478433 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" podStartSLOduration=1.478416247 podStartE2EDuration="1.478416247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:31:32.475799587 +0000 UTC m=+537.104753555" watchObservedRunningTime="2026-07-06 19:31:32.478416247 +0000 UTC m=+537.107370212" Jul 06 19:31:32.616335 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:32.616294 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:32.621235 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:32.621204 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:33.459466 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:33.459434 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:33.460614 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:33.460594 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:31:55.266920 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:55.266885 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-xc6pb"] Jul 06 19:31:55.289477 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:55.289451 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-xc6pb" Jul 06 19:31:55.309431 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:55.309190 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 06 19:31:55.311373 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:55.311348 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-xc6pb"] Jul 06 19:31:55.314595 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:55.314574 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m2d6z\"" Jul 06 19:31:55.400002 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:55.399966 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c9acfc53-44f0-4ccd-9768-7d17fb00394d-data\") pod \"postgres-69fb4594cf-xc6pb\" (UID: \"c9acfc53-44f0-4ccd-9768-7d17fb00394d\") " pod="opendatahub/postgres-69fb4594cf-xc6pb" Jul 06 19:31:55.400002 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:55.400000 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvw5q\" (UniqueName: \"kubernetes.io/projected/c9acfc53-44f0-4ccd-9768-7d17fb00394d-kube-api-access-xvw5q\") pod \"postgres-69fb4594cf-xc6pb\" (UID: \"c9acfc53-44f0-4ccd-9768-7d17fb00394d\") " pod="opendatahub/postgres-69fb4594cf-xc6pb" Jul 06 19:31:55.501575 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:55.501531 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c9acfc53-44f0-4ccd-9768-7d17fb00394d-data\") pod \"postgres-69fb4594cf-xc6pb\" (UID: \"c9acfc53-44f0-4ccd-9768-7d17fb00394d\") " pod="opendatahub/postgres-69fb4594cf-xc6pb" Jul 06 19:31:55.501575 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:55.501577 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvw5q\" (UniqueName: \"kubernetes.io/projected/c9acfc53-44f0-4ccd-9768-7d17fb00394d-kube-api-access-xvw5q\") pod \"postgres-69fb4594cf-xc6pb\" (UID: \"c9acfc53-44f0-4ccd-9768-7d17fb00394d\") " pod="opendatahub/postgres-69fb4594cf-xc6pb" Jul 06 19:31:55.501969 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:55.501949 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c9acfc53-44f0-4ccd-9768-7d17fb00394d-data\") pod \"postgres-69fb4594cf-xc6pb\" (UID: \"c9acfc53-44f0-4ccd-9768-7d17fb00394d\") " pod="opendatahub/postgres-69fb4594cf-xc6pb" Jul 06 19:31:55.521786 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:55.521758 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvw5q\" (UniqueName: \"kubernetes.io/projected/c9acfc53-44f0-4ccd-9768-7d17fb00394d-kube-api-access-xvw5q\") pod \"postgres-69fb4594cf-xc6pb\" (UID: \"c9acfc53-44f0-4ccd-9768-7d17fb00394d\") " pod="opendatahub/postgres-69fb4594cf-xc6pb" Jul 06 19:31:55.600200 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:55.600166 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-xc6pb" Jul 06 19:31:55.755248 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:55.755214 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-xc6pb"] Jul 06 19:31:55.758433 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:31:55.758403 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9acfc53_44f0_4ccd_9768_7d17fb00394d.slice/crio-50ea176b61cdfe74f5d4a437759ca10257e217e511ce885e88a59f2144c5d8f5 WatchSource:0}: Error finding container 50ea176b61cdfe74f5d4a437759ca10257e217e511ce885e88a59f2144c5d8f5: Status 404 returned error can't find the container with id 50ea176b61cdfe74f5d4a437759ca10257e217e511ce885e88a59f2144c5d8f5 Jul 06 19:31:56.582440 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:31:56.582395 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-xc6pb" event={"ID":"c9acfc53-44f0-4ccd-9768-7d17fb00394d","Type":"ContainerStarted","Data":"50ea176b61cdfe74f5d4a437759ca10257e217e511ce885e88a59f2144c5d8f5"} Jul 06 19:32:00.606059 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:00.605968 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-xc6pb" event={"ID":"c9acfc53-44f0-4ccd-9768-7d17fb00394d","Type":"ContainerStarted","Data":"a8eba75027c427834899436b3f920a5a3610bf34c37b31725cf941ebd68ff7b9"} Jul 06 19:32:00.606059 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:00.606035 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-xc6pb" Jul 06 19:32:06.640810 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:06.640771 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-xc6pb" Jul 06 19:32:06.672711 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:06.672644 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-xc6pb" podStartSLOduration=7.089079658 podStartE2EDuration="11.672626417s" totalImagesPullingTime="4.583546759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:31:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:00.631220665 +0000 UTC m=+565.260174631" watchObservedRunningTime="2026-07-06 19:32:06.672626417 +0000 UTC m=+571.301580383" Jul 06 19:32:07.767219 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:07.767184 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f785c4d58-pqvld"] Jul 06 19:32:07.774458 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:07.774432 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f785c4d58-pqvld" Jul 06 19:32:07.776890 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:07.776870 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-j85kz\"" Jul 06 19:32:07.777561 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:07.777543 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 19:32:07.785779 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:07.785755 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f785c4d58-pqvld"] Jul 06 19:32:07.819273 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:07.819232 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7eb86db9-5e48-402a-b69d-f807ee491609-tls-cert\") pod \"authorino-5f785c4d58-pqvld\" (UID: \"7eb86db9-5e48-402a-b69d-f807ee491609\") " pod="kuadrant-system/authorino-5f785c4d58-pqvld" Jul 06 19:32:07.819468 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:07.819317 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qk6f\" (UniqueName: \"kubernetes.io/projected/7eb86db9-5e48-402a-b69d-f807ee491609-kube-api-access-6qk6f\") pod \"authorino-5f785c4d58-pqvld\" (UID: \"7eb86db9-5e48-402a-b69d-f807ee491609\") " pod="kuadrant-system/authorino-5f785c4d58-pqvld" Jul 06 19:32:07.919963 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:07.919923 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7eb86db9-5e48-402a-b69d-f807ee491609-tls-cert\") pod \"authorino-5f785c4d58-pqvld\" (UID: \"7eb86db9-5e48-402a-b69d-f807ee491609\") " pod="kuadrant-system/authorino-5f785c4d58-pqvld" Jul 06 19:32:07.920123 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:07.920006 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qk6f\" (UniqueName: \"kubernetes.io/projected/7eb86db9-5e48-402a-b69d-f807ee491609-kube-api-access-6qk6f\") pod \"authorino-5f785c4d58-pqvld\" (UID: \"7eb86db9-5e48-402a-b69d-f807ee491609\") " pod="kuadrant-system/authorino-5f785c4d58-pqvld" Jul 06 19:32:07.922351 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:07.922318 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7eb86db9-5e48-402a-b69d-f807ee491609-tls-cert\") pod \"authorino-5f785c4d58-pqvld\" (UID: \"7eb86db9-5e48-402a-b69d-f807ee491609\") " pod="kuadrant-system/authorino-5f785c4d58-pqvld" Jul 06 19:32:07.930331 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:07.930304 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qk6f\" (UniqueName: \"kubernetes.io/projected/7eb86db9-5e48-402a-b69d-f807ee491609-kube-api-access-6qk6f\") pod \"authorino-5f785c4d58-pqvld\" (UID: \"7eb86db9-5e48-402a-b69d-f807ee491609\") " pod="kuadrant-system/authorino-5f785c4d58-pqvld" Jul 06 19:32:08.084544 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:08.084504 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f785c4d58-pqvld" Jul 06 19:32:08.236288 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:08.236251 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f785c4d58-pqvld"] Jul 06 19:32:08.239937 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:32:08.239908 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7eb86db9_5e48_402a_b69d_f807ee491609.slice/crio-6c381d5b2c52300d8cbd98458dc562b33b94c868d14905b3dd09f6da2a6677fa WatchSource:0}: Error finding container 6c381d5b2c52300d8cbd98458dc562b33b94c868d14905b3dd09f6da2a6677fa: Status 404 returned error can't find the container with id 6c381d5b2c52300d8cbd98458dc562b33b94c868d14905b3dd09f6da2a6677fa Jul 06 19:32:08.646964 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:08.646924 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f785c4d58-pqvld" event={"ID":"7eb86db9-5e48-402a-b69d-f807ee491609","Type":"ContainerStarted","Data":"6c381d5b2c52300d8cbd98458dc562b33b94c868d14905b3dd09f6da2a6677fa"} Jul 06 19:32:12.673145 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:12.673101 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f785c4d58-pqvld" event={"ID":"7eb86db9-5e48-402a-b69d-f807ee491609","Type":"ContainerStarted","Data":"71dac4084619fa14d277f2e3d24930c27c4f0ac9bfa4a5ac49b6ea4a5afda3b4"} Jul 06 19:32:12.690815 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:12.690752 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f785c4d58-pqvld" podStartSLOduration=2.159982533 podStartE2EDuration="5.69073142s" totalImagesPullingTime="3.530748887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:12.689928377 +0000 UTC m=+577.318882343" watchObservedRunningTime="2026-07-06 19:32:12.69073142 +0000 UTC m=+577.319685386" Jul 06 19:32:35.810496 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:35.810466 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 19:32:35.812989 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:35.812968 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 19:32:57.759453 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:57.759373 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-45k9w"] Jul 06 19:32:57.764619 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:57.764583 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-45k9w" Jul 06 19:32:57.767997 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:57.767975 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 19:32:57.767997 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:57.767994 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pgc25\"" Jul 06 19:32:57.773898 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:57.773872 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-45k9w"] Jul 06 19:32:57.880378 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:57.880349 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e41d8b1-2901-40df-bcc1-9314078eebae-plugins-config-volume\") pod \"payload-processing-58f86c7798-45k9w\" (UID: \"2e41d8b1-2901-40df-bcc1-9314078eebae\") " pod="openshift-ingress/payload-processing-58f86c7798-45k9w" Jul 06 19:32:57.880548 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:57.880402 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvp7x\" (UniqueName: \"kubernetes.io/projected/2e41d8b1-2901-40df-bcc1-9314078eebae-kube-api-access-hvp7x\") pod \"payload-processing-58f86c7798-45k9w\" (UID: \"2e41d8b1-2901-40df-bcc1-9314078eebae\") " pod="openshift-ingress/payload-processing-58f86c7798-45k9w" Jul 06 19:32:57.981381 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:57.981340 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e41d8b1-2901-40df-bcc1-9314078eebae-plugins-config-volume\") pod \"payload-processing-58f86c7798-45k9w\" (UID: \"2e41d8b1-2901-40df-bcc1-9314078eebae\") " pod="openshift-ingress/payload-processing-58f86c7798-45k9w" Jul 06 19:32:57.981551 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:57.981419 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvp7x\" (UniqueName: \"kubernetes.io/projected/2e41d8b1-2901-40df-bcc1-9314078eebae-kube-api-access-hvp7x\") pod \"payload-processing-58f86c7798-45k9w\" (UID: \"2e41d8b1-2901-40df-bcc1-9314078eebae\") " pod="openshift-ingress/payload-processing-58f86c7798-45k9w" Jul 06 19:32:57.982104 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:57.982075 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e41d8b1-2901-40df-bcc1-9314078eebae-plugins-config-volume\") pod \"payload-processing-58f86c7798-45k9w\" (UID: \"2e41d8b1-2901-40df-bcc1-9314078eebae\") " pod="openshift-ingress/payload-processing-58f86c7798-45k9w" Jul 06 19:32:57.991094 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:57.991061 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvp7x\" (UniqueName: \"kubernetes.io/projected/2e41d8b1-2901-40df-bcc1-9314078eebae-kube-api-access-hvp7x\") pod \"payload-processing-58f86c7798-45k9w\" (UID: \"2e41d8b1-2901-40df-bcc1-9314078eebae\") " pod="openshift-ingress/payload-processing-58f86c7798-45k9w" Jul 06 19:32:58.078837 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:58.078798 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-45k9w" Jul 06 19:32:58.273033 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:58.272997 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-45k9w"] Jul 06 19:32:58.277575 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:32:58.277538 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e41d8b1_2901_40df_bcc1_9314078eebae.slice/crio-b194a9c7298b58d04cccbc544e113fa26189837275ad89a4a5b4203d5c0592d0 WatchSource:0}: Error finding container b194a9c7298b58d04cccbc544e113fa26189837275ad89a4a5b4203d5c0592d0: Status 404 returned error can't find the container with id b194a9c7298b58d04cccbc544e113fa26189837275ad89a4a5b4203d5c0592d0 Jul 06 19:32:58.911377 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:58.911339 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-45k9w" event={"ID":"2e41d8b1-2901-40df-bcc1-9314078eebae","Type":"ContainerStarted","Data":"b194a9c7298b58d04cccbc544e113fa26189837275ad89a4a5b4203d5c0592d0"} Jul 06 19:32:59.918111 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:59.918072 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-45k9w" event={"ID":"2e41d8b1-2901-40df-bcc1-9314078eebae","Type":"ContainerStarted","Data":"eb0452afbd5820b28369f7c14ab7a46e24107fb61f620b2e1e6cfe74398034d1"} Jul 06 19:32:59.918510 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:59.918191 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-45k9w" Jul 06 19:32:59.942114 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:32:59.942037 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-45k9w" podStartSLOduration=1.42634718 podStartE2EDuration="2.94201178s" totalImagesPullingTime="1.5156646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:59.937865472 +0000 UTC m=+624.566819440" watchObservedRunningTime="2026-07-06 19:32:59.94201178 +0000 UTC m=+624.570965747" Jul 06 19:33:04.500903 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:04.500861 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-nxb4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 19:33:04.501282 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:04.500928 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" podUID="639168ba-36ec-47a3-a023-6135f27dd8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 19:33:05.950565 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:05.950529 2578 generic.go:356] "Generic (PLEG): container finished" podID="639168ba-36ec-47a3-a023-6135f27dd8d9" containerID="4489cb6ea53d92073eff97ea249973f56cb06cbdfe146e666943323bfc368209" exitCode=0 Jul 06 19:33:05.951157 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:05.950586 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" event={"ID":"639168ba-36ec-47a3-a023-6135f27dd8d9","Type":"ContainerDied","Data":"4489cb6ea53d92073eff97ea249973f56cb06cbdfe146e666943323bfc368209"} Jul 06 19:33:05.951157 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:05.951072 2578 scope.go:117] "RemoveContainer" containerID="4489cb6ea53d92073eff97ea249973f56cb06cbdfe146e666943323bfc368209" Jul 06 19:33:05.952153 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:05.952122 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:33:05.952208 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:05.952191 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:33:05.952242 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:05.952221 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:33:06.958778 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:06.958736 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" event={"ID":"639168ba-36ec-47a3-a023-6135f27dd8d9","Type":"ContainerStarted","Data":"414770e79449551e3e366295b9183c172cb7ba4fa672e069069df11d2f7fa84d"} Jul 06 19:33:07.616798 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:07.616750 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:33:07.617046 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:07.617022 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-nxb4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 06 19:33:07.617110 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:07.617075 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" podUID="639168ba-36ec-47a3-a023-6135f27dd8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 06 19:33:08.616072 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:08.616034 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-nxb4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 06 19:33:08.616466 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:08.616098 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" podUID="639168ba-36ec-47a3-a023-6135f27dd8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 06 19:33:09.616112 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:09.616082 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-nxb4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 06 19:33:09.616496 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:09.616136 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" podUID="639168ba-36ec-47a3-a023-6135f27dd8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 06 19:33:10.616051 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:10.616011 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-nxb4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 06 19:33:10.616250 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:10.616071 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" podUID="639168ba-36ec-47a3-a023-6135f27dd8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 06 19:33:10.926460 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:10.926381 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-45k9w" Jul 06 19:33:11.619846 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:11.619819 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:33:11.620215 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:11.620197 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:33:11.621023 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:11.621007 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nxb4p" Jul 06 19:33:45.952937 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:45.952907 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl"] Jul 06 19:33:45.956802 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:45.956780 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:45.962238 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:45.962206 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 06 19:33:45.962238 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:45.962229 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-spm8h\"" Jul 06 19:33:45.962732 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:45.962715 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 19:33:45.977460 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:45.977435 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 19:33:45.986599 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:45.986578 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl"] Jul 06 19:33:46.018247 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.018212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9122cc34-84f5-4484-8e9a-981cdf612efb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl\" (UID: \"9122cc34-84f5-4484-8e9a-981cdf612efb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:46.018247 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.018251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9122cc34-84f5-4484-8e9a-981cdf612efb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl\" (UID: \"9122cc34-84f5-4484-8e9a-981cdf612efb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:46.018491 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.018274 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9122cc34-84f5-4484-8e9a-981cdf612efb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl\" (UID: \"9122cc34-84f5-4484-8e9a-981cdf612efb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:46.018491 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.018402 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m586\" (UniqueName: \"kubernetes.io/projected/9122cc34-84f5-4484-8e9a-981cdf612efb-kube-api-access-9m586\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl\" (UID: \"9122cc34-84f5-4484-8e9a-981cdf612efb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:46.018491 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.018446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9122cc34-84f5-4484-8e9a-981cdf612efb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl\" (UID: \"9122cc34-84f5-4484-8e9a-981cdf612efb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:46.018491 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.018488 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9122cc34-84f5-4484-8e9a-981cdf612efb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl\" (UID: \"9122cc34-84f5-4484-8e9a-981cdf612efb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:46.119281 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.119247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m586\" (UniqueName: \"kubernetes.io/projected/9122cc34-84f5-4484-8e9a-981cdf612efb-kube-api-access-9m586\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl\" (UID: \"9122cc34-84f5-4484-8e9a-981cdf612efb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:46.119485 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.119293 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9122cc34-84f5-4484-8e9a-981cdf612efb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl\" (UID: \"9122cc34-84f5-4484-8e9a-981cdf612efb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:46.119485 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.119336 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9122cc34-84f5-4484-8e9a-981cdf612efb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl\" (UID: \"9122cc34-84f5-4484-8e9a-981cdf612efb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:46.119485 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.119446 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9122cc34-84f5-4484-8e9a-981cdf612efb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl\" (UID: \"9122cc34-84f5-4484-8e9a-981cdf612efb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:46.119485 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.119481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9122cc34-84f5-4484-8e9a-981cdf612efb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl\" (UID: \"9122cc34-84f5-4484-8e9a-981cdf612efb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:46.119735 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.119508 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9122cc34-84f5-4484-8e9a-981cdf612efb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl\" (UID: \"9122cc34-84f5-4484-8e9a-981cdf612efb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:46.119910 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.119872 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9122cc34-84f5-4484-8e9a-981cdf612efb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl\" (UID: \"9122cc34-84f5-4484-8e9a-981cdf612efb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:46.119984 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.119886 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9122cc34-84f5-4484-8e9a-981cdf612efb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl\" (UID: \"9122cc34-84f5-4484-8e9a-981cdf612efb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:46.119984 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.119937 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9122cc34-84f5-4484-8e9a-981cdf612efb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl\" (UID: \"9122cc34-84f5-4484-8e9a-981cdf612efb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:46.121675 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.121652 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9122cc34-84f5-4484-8e9a-981cdf612efb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl\" (UID: \"9122cc34-84f5-4484-8e9a-981cdf612efb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:46.121992 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.121973 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9122cc34-84f5-4484-8e9a-981cdf612efb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl\" (UID: \"9122cc34-84f5-4484-8e9a-981cdf612efb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:46.134177 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.134154 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m586\" (UniqueName: \"kubernetes.io/projected/9122cc34-84f5-4484-8e9a-981cdf612efb-kube-api-access-9m586\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl\" (UID: \"9122cc34-84f5-4484-8e9a-981cdf612efb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:46.266631 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.266552 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:46.430572 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.430537 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl"] Jul 06 19:33:46.434167 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:33:46.434141 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9122cc34_84f5_4484_8e9a_981cdf612efb.slice/crio-013eb9950d101b912a3abac6b545d476ec6e7b76d546e53727bb4a553ce88bdf WatchSource:0}: Error finding container 013eb9950d101b912a3abac6b545d476ec6e7b76d546e53727bb4a553ce88bdf: Status 404 returned error can't find the container with id 013eb9950d101b912a3abac6b545d476ec6e7b76d546e53727bb4a553ce88bdf Jul 06 19:33:46.435521 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:46.435502 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:33:47.168507 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:47.168464 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" event={"ID":"9122cc34-84f5-4484-8e9a-981cdf612efb","Type":"ContainerStarted","Data":"013eb9950d101b912a3abac6b545d476ec6e7b76d546e53727bb4a553ce88bdf"} Jul 06 19:33:48.174978 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:48.174941 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" event={"ID":"9122cc34-84f5-4484-8e9a-981cdf612efb","Type":"ContainerStarted","Data":"3bf2420ec25bdb4d225638686fc01f25713c814f572a5873ceb4a40e036b520d"} Jul 06 19:33:48.203747 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:48.203665 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" podStartSLOduration=2.1336976930000002 podStartE2EDuration="3.203644009s" totalImagesPullingTime="1.069946316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:48.200542649 +0000 UTC m=+672.829496616" watchObservedRunningTime="2026-07-06 19:33:48.203644009 +0000 UTC m=+672.832597975" Jul 06 19:33:56.267556 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:56.267516 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:56.268011 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:56.267616 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:56.280736 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:56.280706 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:33:57.235646 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:33:57.235613 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl" Jul 06 19:35:31.207422 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:31.207382 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f785c4d58-pqvld"] Jul 06 19:35:31.207959 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:31.207597 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5f785c4d58-pqvld" podUID="7eb86db9-5e48-402a-b69d-f807ee491609" containerName="authorino" containerID="cri-o://71dac4084619fa14d277f2e3d24930c27c4f0ac9bfa4a5ac49b6ea4a5afda3b4" gracePeriod=30 Jul 06 19:35:31.467656 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:31.467583 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f785c4d58-pqvld" Jul 06 19:35:31.562995 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:31.562960 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qk6f\" (UniqueName: \"kubernetes.io/projected/7eb86db9-5e48-402a-b69d-f807ee491609-kube-api-access-6qk6f\") pod \"7eb86db9-5e48-402a-b69d-f807ee491609\" (UID: \"7eb86db9-5e48-402a-b69d-f807ee491609\") " Jul 06 19:35:31.563155 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:31.563049 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7eb86db9-5e48-402a-b69d-f807ee491609-tls-cert\") pod \"7eb86db9-5e48-402a-b69d-f807ee491609\" (UID: \"7eb86db9-5e48-402a-b69d-f807ee491609\") " Jul 06 19:35:31.565123 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:31.565093 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb86db9-5e48-402a-b69d-f807ee491609-kube-api-access-6qk6f" (OuterVolumeSpecName: "kube-api-access-6qk6f") pod "7eb86db9-5e48-402a-b69d-f807ee491609" (UID: "7eb86db9-5e48-402a-b69d-f807ee491609"). InnerVolumeSpecName "kube-api-access-6qk6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:35:31.574422 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:31.574394 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eb86db9-5e48-402a-b69d-f807ee491609-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "7eb86db9-5e48-402a-b69d-f807ee491609" (UID: "7eb86db9-5e48-402a-b69d-f807ee491609"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:35:31.664520 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:31.664485 2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7eb86db9-5e48-402a-b69d-f807ee491609-tls-cert\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:35:31.664520 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:31.664517 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qk6f\" (UniqueName: \"kubernetes.io/projected/7eb86db9-5e48-402a-b69d-f807ee491609-kube-api-access-6qk6f\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:35:31.717051 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:31.717019 2578 generic.go:356] "Generic (PLEG): container finished" podID="7eb86db9-5e48-402a-b69d-f807ee491609" containerID="71dac4084619fa14d277f2e3d24930c27c4f0ac9bfa4a5ac49b6ea4a5afda3b4" exitCode=0 Jul 06 19:35:31.717246 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:31.717070 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f785c4d58-pqvld" event={"ID":"7eb86db9-5e48-402a-b69d-f807ee491609","Type":"ContainerDied","Data":"71dac4084619fa14d277f2e3d24930c27c4f0ac9bfa4a5ac49b6ea4a5afda3b4"} Jul 06 19:35:31.717246 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:31.717076 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f785c4d58-pqvld" Jul 06 19:35:31.717246 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:31.717106 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f785c4d58-pqvld" event={"ID":"7eb86db9-5e48-402a-b69d-f807ee491609","Type":"ContainerDied","Data":"6c381d5b2c52300d8cbd98458dc562b33b94c868d14905b3dd09f6da2a6677fa"} Jul 06 19:35:31.717246 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:31.717125 2578 scope.go:117] "RemoveContainer" containerID="71dac4084619fa14d277f2e3d24930c27c4f0ac9bfa4a5ac49b6ea4a5afda3b4" Jul 06 19:35:31.726161 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:31.726140 2578 scope.go:117] "RemoveContainer" containerID="71dac4084619fa14d277f2e3d24930c27c4f0ac9bfa4a5ac49b6ea4a5afda3b4" Jul 06 19:35:31.726427 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:35:31.726404 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71dac4084619fa14d277f2e3d24930c27c4f0ac9bfa4a5ac49b6ea4a5afda3b4\": container with ID starting with 71dac4084619fa14d277f2e3d24930c27c4f0ac9bfa4a5ac49b6ea4a5afda3b4 not found: ID does not exist" containerID="71dac4084619fa14d277f2e3d24930c27c4f0ac9bfa4a5ac49b6ea4a5afda3b4" Jul 06 19:35:31.726501 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:31.726439 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71dac4084619fa14d277f2e3d24930c27c4f0ac9bfa4a5ac49b6ea4a5afda3b4"} err="failed to get container status \"71dac4084619fa14d277f2e3d24930c27c4f0ac9bfa4a5ac49b6ea4a5afda3b4\": rpc error: code = NotFound desc = could not find container \"71dac4084619fa14d277f2e3d24930c27c4f0ac9bfa4a5ac49b6ea4a5afda3b4\": container with ID starting with 71dac4084619fa14d277f2e3d24930c27c4f0ac9bfa4a5ac49b6ea4a5afda3b4 not found: ID does not exist" Jul 06 19:35:31.749431 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:31.749390 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f785c4d58-pqvld"] Jul 06 19:35:31.780707 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:31.780664 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f785c4d58-pqvld"] Jul 06 19:35:31.876175 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:31.876142 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eb86db9-5e48-402a-b69d-f807ee491609" path="/var/lib/kubelet/pods/7eb86db9-5e48-402a-b69d-f807ee491609/volumes" Jul 06 19:35:53.517794 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:53.517668 2578 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" podUID="f55184b1-bd4b-4da5-9f7a-d6099ed678ab" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.31:8081/healthz\": dial tcp 10.132.0.31:8081: connect: connection refused" Jul 06 19:35:53.666335 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:53.666290 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" podUID="f55184b1-bd4b-4da5-9f7a-d6099ed678ab" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.31:8081/readyz\": dial tcp 10.132.0.31:8081: connect: connection refused" Jul 06 19:35:53.832056 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:53.832023 2578 generic.go:356] "Generic (PLEG): container finished" podID="f55184b1-bd4b-4da5-9f7a-d6099ed678ab" containerID="30ae76a8df34415c68920d893fb1a6d73aae5c5bb4ddc0fc5f600b867b394c19" exitCode=1 Jul 06 19:35:53.832239 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:53.832096 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" event={"ID":"f55184b1-bd4b-4da5-9f7a-d6099ed678ab","Type":"ContainerDied","Data":"30ae76a8df34415c68920d893fb1a6d73aae5c5bb4ddc0fc5f600b867b394c19"} Jul 06 19:35:53.832503 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:53.832486 2578 scope.go:117] "RemoveContainer" containerID="30ae76a8df34415c68920d893fb1a6d73aae5c5bb4ddc0fc5f600b867b394c19" Jul 06 19:35:54.838801 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.838764 2578 generic.go:356] "Generic (PLEG): container finished" podID="85caf392-2962-4d96-8ae0-59dbf3366abe" containerID="6cc1fcfc63a5cb0661d419c23e02e8c07c2efa578f29b555f8b367336c2fc2af" exitCode=1 Jul 06 19:35:54.839282 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.838837 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" event={"ID":"85caf392-2962-4d96-8ae0-59dbf3366abe","Type":"ContainerDied","Data":"6cc1fcfc63a5cb0661d419c23e02e8c07c2efa578f29b555f8b367336c2fc2af"} Jul 06 19:35:54.839282 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.838897 2578 scope.go:117] "RemoveContainer" containerID="dae5d7722ca62b5791f0079e17a53145d49f58a6b1dc33d14959760a0abb608d" Jul 06 19:35:54.839282 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.839240 2578 scope.go:117] "RemoveContainer" containerID="6cc1fcfc63a5cb0661d419c23e02e8c07c2efa578f29b555f8b367336c2fc2af" Jul 06 19:35:54.839456 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:35:54.839435 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=odh-model-controller-7998c67dc4-kkjcw_opendatahub(85caf392-2962-4d96-8ae0-59dbf3366abe)\"" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" podUID="85caf392-2962-4d96-8ae0-59dbf3366abe" Jul 06 19:35:54.840823 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.840803 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" event={"ID":"f55184b1-bd4b-4da5-9f7a-d6099ed678ab","Type":"ContainerStarted","Data":"3b91808346c79bc006dafe07af2c6eaece5805d1c0bd80fce84107c2b3c90e81"} Jul 06 19:35:54.840982 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.840966 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:35:54.842243 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.842224 2578 generic.go:356] "Generic (PLEG): container finished" podID="40393be6-bd01-4617-ace4-5cbdfcd186fd" containerID="e8adacd47c5619e2638d920f44a2af81ab817605a3e9faa1368c237893d034ce" exitCode=1 Jul 06 19:35:54.842314 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.842296 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" event={"ID":"40393be6-bd01-4617-ace4-5cbdfcd186fd","Type":"ContainerDied","Data":"e8adacd47c5619e2638d920f44a2af81ab817605a3e9faa1368c237893d034ce"} Jul 06 19:35:54.842571 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.842552 2578 scope.go:117] "RemoveContainer" containerID="e8adacd47c5619e2638d920f44a2af81ab817605a3e9faa1368c237893d034ce" Jul 06 19:35:54.843911 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.843894 2578 generic.go:356] "Generic (PLEG): container finished" podID="18778df4-9ca2-41fe-a983-e69f2d564dd2" containerID="67fb9e8903cc9017727d9e8717a3ff14a41133cdda1c9288588ffcdc83acb962" exitCode=1 Jul 06 19:35:54.844002 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.843964 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" event={"ID":"18778df4-9ca2-41fe-a983-e69f2d564dd2","Type":"ContainerDied","Data":"67fb9e8903cc9017727d9e8717a3ff14a41133cdda1c9288588ffcdc83acb962"} Jul 06 19:35:54.844247 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.844230 2578 scope.go:117] "RemoveContainer" containerID="67fb9e8903cc9017727d9e8717a3ff14a41133cdda1c9288588ffcdc83acb962" Jul 06 19:35:54.845975 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.845956 2578 generic.go:356] "Generic (PLEG): container finished" podID="f1f2d36b-4239-45fa-a6d7-85c316bd426e" containerID="8256b2550654e5c34e69b16380fdb1c7d2a644b9b56f133f6c13b9738c4f339b" exitCode=1 Jul 06 19:35:54.846075 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.846014 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" event={"ID":"f1f2d36b-4239-45fa-a6d7-85c316bd426e","Type":"ContainerDied","Data":"8256b2550654e5c34e69b16380fdb1c7d2a644b9b56f133f6c13b9738c4f339b"} Jul 06 19:35:54.846394 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.846370 2578 scope.go:117] "RemoveContainer" containerID="8256b2550654e5c34e69b16380fdb1c7d2a644b9b56f133f6c13b9738c4f339b" Jul 06 19:35:54.848034 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.848013 2578 generic.go:356] "Generic (PLEG): container finished" podID="7f55d1c1-4236-45e8-812a-68f2b9015ff6" containerID="fcbe2dbbb877039acd663056df24753798471b9700ebbed0eba515b535f8d27a" exitCode=1 Jul 06 19:35:54.848146 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.848089 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" event={"ID":"7f55d1c1-4236-45e8-812a-68f2b9015ff6","Type":"ContainerDied","Data":"fcbe2dbbb877039acd663056df24753798471b9700ebbed0eba515b535f8d27a"} Jul 06 19:35:54.848393 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.848375 2578 scope.go:117] "RemoveContainer" containerID="fcbe2dbbb877039acd663056df24753798471b9700ebbed0eba515b535f8d27a" Jul 06 19:35:54.849779 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.849759 2578 generic.go:356] "Generic (PLEG): container finished" podID="fc0a6e65-42be-43eb-8a55-363e166ecc16" containerID="a621f2ae25ab18ce1e39c9bdf6a04cc658bada680dfa5e5f8dffbdcc84b924a4" exitCode=1 Jul 06 19:35:54.849912 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.849831 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" event={"ID":"fc0a6e65-42be-43eb-8a55-363e166ecc16","Type":"ContainerDied","Data":"a621f2ae25ab18ce1e39c9bdf6a04cc658bada680dfa5e5f8dffbdcc84b924a4"} Jul 06 19:35:54.850056 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:54.850044 2578 scope.go:117] "RemoveContainer" containerID="a621f2ae25ab18ce1e39c9bdf6a04cc658bada680dfa5e5f8dffbdcc84b924a4" Jul 06 19:35:55.481019 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:55.480930 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:35:55.857748 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:55.857659 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" event={"ID":"f1f2d36b-4239-45fa-a6d7-85c316bd426e","Type":"ContainerStarted","Data":"545853f6be91f627ce1bde6d1bced56d13f20b8295054db42729a507835c6066"} Jul 06 19:35:55.858156 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:55.857832 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:35:55.859478 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:55.859460 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" event={"ID":"7f55d1c1-4236-45e8-812a-68f2b9015ff6","Type":"ContainerStarted","Data":"eb2756f794802cef8dc0f896eec3babdc853529c6836b5a5dae2a3845c1a52d8"} Jul 06 19:35:55.859652 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:55.859639 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" Jul 06 19:35:55.861213 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:55.861193 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" event={"ID":"fc0a6e65-42be-43eb-8a55-363e166ecc16","Type":"ContainerStarted","Data":"cc8e8756489398d4a62650cef6f4de6d384b86265df1f3151868d99e5d0d6115"} Jul 06 19:35:55.861358 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:55.861345 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" Jul 06 19:35:55.864054 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:55.864028 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" event={"ID":"40393be6-bd01-4617-ace4-5cbdfcd186fd","Type":"ContainerStarted","Data":"b428257107c2dc599015f76664541decf160b6e2f5a90c4f31f302b42e2ac281"} Jul 06 19:35:55.864202 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:55.864187 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:35:55.865920 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:55.865900 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" event={"ID":"18778df4-9ca2-41fe-a983-e69f2d564dd2","Type":"ContainerStarted","Data":"af2d801c1a24c000d36d46b44b0cabdadfbc36698487048f9a199d03fdd8e6fa"} Jul 06 19:35:55.866110 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:35:55.866097 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" Jul 06 19:36:00.894506 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:36:00.894476 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:36:00.894954 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:36:00.894829 2578 scope.go:117] "RemoveContainer" containerID="6cc1fcfc63a5cb0661d419c23e02e8c07c2efa578f29b555f8b367336c2fc2af" Jul 06 19:36:00.895025 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:36:00.895008 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=odh-model-controller-7998c67dc4-kkjcw_opendatahub(85caf392-2962-4d96-8ae0-59dbf3366abe)\"" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" podUID="85caf392-2962-4d96-8ae0-59dbf3366abe" Jul 06 19:36:02.928257 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:36:02.928214 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:36:02.928642 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:36:02.928633 2578 scope.go:117] "RemoveContainer" containerID="6cc1fcfc63a5cb0661d419c23e02e8c07c2efa578f29b555f8b367336c2fc2af" Jul 06 19:36:02.928860 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:36:02.928840 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=odh-model-controller-7998c67dc4-kkjcw_opendatahub(85caf392-2962-4d96-8ae0-59dbf3366abe)\"" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" podUID="85caf392-2962-4d96-8ae0-59dbf3366abe" Jul 06 19:36:06.873320 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:36:06.873283 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:36:06.873849 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:36:06.873374 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" Jul 06 19:36:06.873849 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:36:06.873493 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" Jul 06 19:36:06.873849 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:36:06.873835 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" Jul 06 19:36:15.873154 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:36:15.873120 2578 scope.go:117] "RemoveContainer" containerID="6cc1fcfc63a5cb0661d419c23e02e8c07c2efa578f29b555f8b367336c2fc2af" Jul 06 19:36:16.980773 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:36:16.980737 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" event={"ID":"85caf392-2962-4d96-8ae0-59dbf3366abe","Type":"ContainerStarted","Data":"a20488db2c13f8bb374cd2ea8d7b3a1118af42c57fe3a25d6d293b8cdb9f8488"} Jul 06 19:36:16.981179 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:36:16.980976 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:36:25.868044 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:36:25.868006 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:36:26.875898 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:36:26.875867 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:36:27.988206 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:36:27.988174 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:37:35.845857 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:37:35.845777 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 19:37:35.845857 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:37:35.845777 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 19:40:40.866914 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:40.866863 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" podUID="f55184b1-bd4b-4da5-9f7a-d6099ed678ab" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.31:8081/readyz\": dial tcp 10.132.0.31:8081: connect: connection refused" Jul 06 19:40:40.894945 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:40.894900 2578 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" podUID="85caf392-2962-4d96-8ae0-59dbf3366abe" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/healthz\": dial tcp 10.132.0.28:8081: connect: connection refused" Jul 06 19:40:41.382063 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.382026 2578 generic.go:356] "Generic (PLEG): container finished" podID="85caf392-2962-4d96-8ae0-59dbf3366abe" containerID="a20488db2c13f8bb374cd2ea8d7b3a1118af42c57fe3a25d6d293b8cdb9f8488" exitCode=1 Jul 06 19:40:41.382232 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.382100 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" event={"ID":"85caf392-2962-4d96-8ae0-59dbf3366abe","Type":"ContainerDied","Data":"a20488db2c13f8bb374cd2ea8d7b3a1118af42c57fe3a25d6d293b8cdb9f8488"} Jul 06 19:40:41.382232 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.382180 2578 scope.go:117] "RemoveContainer" containerID="6cc1fcfc63a5cb0661d419c23e02e8c07c2efa578f29b555f8b367336c2fc2af" Jul 06 19:40:41.382528 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.382500 2578 scope.go:117] "RemoveContainer" containerID="a20488db2c13f8bb374cd2ea8d7b3a1118af42c57fe3a25d6d293b8cdb9f8488" Jul 06 19:40:41.382762 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:40:41.382743 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=odh-model-controller-7998c67dc4-kkjcw_opendatahub(85caf392-2962-4d96-8ae0-59dbf3366abe)\"" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" podUID="85caf392-2962-4d96-8ae0-59dbf3366abe" Jul 06 19:40:41.384093 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.384070 2578 generic.go:356] "Generic (PLEG): container finished" podID="f55184b1-bd4b-4da5-9f7a-d6099ed678ab" containerID="3b91808346c79bc006dafe07af2c6eaece5805d1c0bd80fce84107c2b3c90e81" exitCode=1 Jul 06 19:40:41.384183 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.384140 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" event={"ID":"f55184b1-bd4b-4da5-9f7a-d6099ed678ab","Type":"ContainerDied","Data":"3b91808346c79bc006dafe07af2c6eaece5805d1c0bd80fce84107c2b3c90e81"} Jul 06 19:40:41.384457 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.384443 2578 scope.go:117] "RemoveContainer" containerID="3b91808346c79bc006dafe07af2c6eaece5805d1c0bd80fce84107c2b3c90e81" Jul 06 19:40:41.384616 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:40:41.384601 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-hrvh5_opendatahub(f55184b1-bd4b-4da5-9f7a-d6099ed678ab)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" podUID="f55184b1-bd4b-4da5-9f7a-d6099ed678ab" Jul 06 19:40:41.385957 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.385941 2578 generic.go:356] "Generic (PLEG): container finished" podID="40393be6-bd01-4617-ace4-5cbdfcd186fd" containerID="b428257107c2dc599015f76664541decf160b6e2f5a90c4f31f302b42e2ac281" exitCode=1 Jul 06 19:40:41.386042 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.386009 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" event={"ID":"40393be6-bd01-4617-ace4-5cbdfcd186fd","Type":"ContainerDied","Data":"b428257107c2dc599015f76664541decf160b6e2f5a90c4f31f302b42e2ac281"} Jul 06 19:40:41.386311 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.386294 2578 scope.go:117] "RemoveContainer" containerID="b428257107c2dc599015f76664541decf160b6e2f5a90c4f31f302b42e2ac281" Jul 06 19:40:41.386494 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:40:41.386474 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pv5d5_opendatahub(40393be6-bd01-4617-ace4-5cbdfcd186fd)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" podUID="40393be6-bd01-4617-ace4-5cbdfcd186fd" Jul 06 19:40:41.387923 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.387907 2578 generic.go:356] "Generic (PLEG): container finished" podID="18778df4-9ca2-41fe-a983-e69f2d564dd2" containerID="af2d801c1a24c000d36d46b44b0cabdadfbc36698487048f9a199d03fdd8e6fa" exitCode=1 Jul 06 19:40:41.388013 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.387980 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" event={"ID":"18778df4-9ca2-41fe-a983-e69f2d564dd2","Type":"ContainerDied","Data":"af2d801c1a24c000d36d46b44b0cabdadfbc36698487048f9a199d03fdd8e6fa"} Jul 06 19:40:41.388259 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.388243 2578 scope.go:117] "RemoveContainer" containerID="af2d801c1a24c000d36d46b44b0cabdadfbc36698487048f9a199d03fdd8e6fa" Jul 06 19:40:41.388417 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:40:41.388398 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=dns-operator-controller-manager-864dd78c4c-6br2w_kuadrant-system(18778df4-9ca2-41fe-a983-e69f2d564dd2)\"" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" podUID="18778df4-9ca2-41fe-a983-e69f2d564dd2" Jul 06 19:40:41.389921 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.389903 2578 generic.go:356] "Generic (PLEG): container finished" podID="7f55d1c1-4236-45e8-812a-68f2b9015ff6" containerID="eb2756f794802cef8dc0f896eec3babdc853529c6836b5a5dae2a3845c1a52d8" exitCode=1 Jul 06 19:40:41.390013 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.389962 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" event={"ID":"7f55d1c1-4236-45e8-812a-68f2b9015ff6","Type":"ContainerDied","Data":"eb2756f794802cef8dc0f896eec3babdc853529c6836b5a5dae2a3845c1a52d8"} Jul 06 19:40:41.390278 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.390258 2578 scope.go:117] "RemoveContainer" containerID="eb2756f794802cef8dc0f896eec3babdc853529c6836b5a5dae2a3845c1a52d8" Jul 06 19:40:41.390451 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:40:41.390434 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=authorino-operator-75cc8d5cb7-ndlnz_kuadrant-system(7f55d1c1-4236-45e8-812a-68f2b9015ff6)\"" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" podUID="7f55d1c1-4236-45e8-812a-68f2b9015ff6" Jul 06 19:40:41.391932 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.391915 2578 generic.go:356] "Generic (PLEG): container finished" podID="fc0a6e65-42be-43eb-8a55-363e166ecc16" containerID="cc8e8756489398d4a62650cef6f4de6d384b86265df1f3151868d99e5d0d6115" exitCode=1 Jul 06 19:40:41.392007 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.391986 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" event={"ID":"fc0a6e65-42be-43eb-8a55-363e166ecc16","Type":"ContainerDied","Data":"cc8e8756489398d4a62650cef6f4de6d384b86265df1f3151868d99e5d0d6115"} Jul 06 19:40:41.392253 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.392241 2578 scope.go:117] "RemoveContainer" containerID="cc8e8756489398d4a62650cef6f4de6d384b86265df1f3151868d99e5d0d6115" Jul 06 19:40:41.392398 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:40:41.392382 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=limitador-operator-controller-manager-859d7b78bd-4lrbz_kuadrant-system(fc0a6e65-42be-43eb-8a55-363e166ecc16)\"" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" podUID="fc0a6e65-42be-43eb-8a55-363e166ecc16" Jul 06 19:40:41.394722 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.394708 2578 scope.go:117] "RemoveContainer" containerID="30ae76a8df34415c68920d893fb1a6d73aae5c5bb4ddc0fc5f600b867b394c19" Jul 06 19:40:41.406874 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.406790 2578 scope.go:117] "RemoveContainer" containerID="e8adacd47c5619e2638d920f44a2af81ab817605a3e9faa1368c237893d034ce" Jul 06 19:40:41.418745 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.418723 2578 scope.go:117] "RemoveContainer" containerID="67fb9e8903cc9017727d9e8717a3ff14a41133cdda1c9288588ffcdc83acb962" Jul 06 19:40:41.430465 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.430441 2578 scope.go:117] "RemoveContainer" containerID="fcbe2dbbb877039acd663056df24753798471b9700ebbed0eba515b535f8d27a" Jul 06 19:40:41.442014 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.441992 2578 scope.go:117] "RemoveContainer" containerID="a621f2ae25ab18ce1e39c9bdf6a04cc658bada680dfa5e5f8dffbdcc84b924a4" Jul 06 19:40:41.875372 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:41.875340 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:40:42.398763 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:42.398737 2578 scope.go:117] "RemoveContainer" containerID="b428257107c2dc599015f76664541decf160b6e2f5a90c4f31f302b42e2ac281" Jul 06 19:40:42.398991 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:40:42.398968 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pv5d5_opendatahub(40393be6-bd01-4617-ace4-5cbdfcd186fd)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" podUID="40393be6-bd01-4617-ace4-5cbdfcd186fd" Jul 06 19:40:42.864793 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:42.864752 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:40:43.410234 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:43.410211 2578 scope.go:117] "RemoveContainer" containerID="b428257107c2dc599015f76664541decf160b6e2f5a90c4f31f302b42e2ac281" Jul 06 19:40:43.410599 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:40:43.410411 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pv5d5_opendatahub(40393be6-bd01-4617-ace4-5cbdfcd186fd)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" podUID="40393be6-bd01-4617-ace4-5cbdfcd186fd" Jul 06 19:40:43.517303 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:43.517266 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:40:43.517652 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:43.517639 2578 scope.go:117] "RemoveContainer" containerID="3b91808346c79bc006dafe07af2c6eaece5805d1c0bd80fce84107c2b3c90e81" Jul 06 19:40:43.517882 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:40:43.517863 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-hrvh5_opendatahub(f55184b1-bd4b-4da5-9f7a-d6099ed678ab)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" podUID="f55184b1-bd4b-4da5-9f7a-d6099ed678ab" Jul 06 19:40:45.867007 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:45.866971 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:40:45.867425 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:45.867357 2578 scope.go:117] "RemoveContainer" containerID="3b91808346c79bc006dafe07af2c6eaece5805d1c0bd80fce84107c2b3c90e81" Jul 06 19:40:45.867564 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:40:45.867544 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-hrvh5_opendatahub(f55184b1-bd4b-4da5-9f7a-d6099ed678ab)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" podUID="f55184b1-bd4b-4da5-9f7a-d6099ed678ab" Jul 06 19:40:46.870861 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:46.870821 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" Jul 06 19:40:46.871364 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:46.870890 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" Jul 06 19:40:46.871364 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:46.870910 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" Jul 06 19:40:46.871364 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:46.871232 2578 scope.go:117] "RemoveContainer" containerID="cc8e8756489398d4a62650cef6f4de6d384b86265df1f3151868d99e5d0d6115" Jul 06 19:40:46.871364 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:46.871294 2578 scope.go:117] "RemoveContainer" containerID="eb2756f794802cef8dc0f896eec3babdc853529c6836b5a5dae2a3845c1a52d8" Jul 06 19:40:46.871558 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:46.871370 2578 scope.go:117] "RemoveContainer" containerID="af2d801c1a24c000d36d46b44b0cabdadfbc36698487048f9a199d03fdd8e6fa" Jul 06 19:40:46.871558 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:40:46.871404 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=limitador-operator-controller-manager-859d7b78bd-4lrbz_kuadrant-system(fc0a6e65-42be-43eb-8a55-363e166ecc16)\"" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" podUID="fc0a6e65-42be-43eb-8a55-363e166ecc16" Jul 06 19:40:46.871558 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:40:46.871440 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=authorino-operator-75cc8d5cb7-ndlnz_kuadrant-system(7f55d1c1-4236-45e8-812a-68f2b9015ff6)\"" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" podUID="7f55d1c1-4236-45e8-812a-68f2b9015ff6" Jul 06 19:40:46.871558 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:40:46.871541 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=dns-operator-controller-manager-864dd78c4c-6br2w_kuadrant-system(18778df4-9ca2-41fe-a983-e69f2d564dd2)\"" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" podUID="18778df4-9ca2-41fe-a983-e69f2d564dd2" Jul 06 19:40:47.985919 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:47.985872 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:40:47.986319 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:47.986227 2578 scope.go:117] "RemoveContainer" containerID="a20488db2c13f8bb374cd2ea8d7b3a1118af42c57fe3a25d6d293b8cdb9f8488" Jul 06 19:40:47.986440 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:40:47.986420 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=odh-model-controller-7998c67dc4-kkjcw_opendatahub(85caf392-2962-4d96-8ae0-59dbf3366abe)\"" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" podUID="85caf392-2962-4d96-8ae0-59dbf3366abe" Jul 06 19:40:49.178319 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:49.178284 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" Jul 06 19:40:49.178855 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:49.178837 2578 scope.go:117] "RemoveContainer" containerID="eb2756f794802cef8dc0f896eec3babdc853529c6836b5a5dae2a3845c1a52d8" Jul 06 19:40:49.179100 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:40:49.179078 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=authorino-operator-75cc8d5cb7-ndlnz_kuadrant-system(7f55d1c1-4236-45e8-812a-68f2b9015ff6)\"" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" podUID="7f55d1c1-4236-45e8-812a-68f2b9015ff6" Jul 06 19:40:56.320605 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:40:56.320547 2578 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"85caf392-2962-4d96-8ae0-59dbf3366abe\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-06T19:40:41Z\\\",\\\"message\\\":\\\"containers with unready status: [manager]\\\",\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-06T19:40:41Z\\\",\\\"message\\\":\\\"containers with unready status: [manager]\\\",\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"64Mi\\\"},\\\"containerID\\\":\\\"cri-o://a20488db2c13f8bb374cd2ea8d7b3a1118af42c57fe3a25d6d293b8cdb9f8488\\\",\\\"image\\\":\\\"quay.io/opendatahub/odh-model-controller:odh-v3.5-EA2\\\",\\\"imageID\\\":\\\"quay.io/opendatahub/odh-model-controller@sha256:1e2397cbb77c598f2eaff55caa3f0ef39f9f499bc7388d4d92be8d2a5d5ff121\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://6cc1fcfc63a5cb0661d419c23e02e8c07c2efa578f29b555f8b367336c2fc2af\\\",\\\"exitCode\\\":1,\\\"finishedAt\\\":\\\"2026-07-06T19:35:54Z\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-07-06T19:29:51Z\\\"}},\\\"name\\\":\\\"manager\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"500m\\\",\\\"memory\\\":\\\"2Gi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"64Mi\\\"}},\\\"restartCount\\\":2,\\\"started\\\":false,\\\"state\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://a20488db2c13f8bb374cd2ea8d7b3a1118af42c57fe3a25d6d293b8cdb9f8488\\\",\\\"exitCode\\\":1,\\\"finishedAt\\\":\\\"2026-07-06T19:40:40Z\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-07-06T19:36:16Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000660000],\\\"uid\\\":1000660000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/tmp/k8s-webhook-server/serving-certs\\\",\\\"name\\\":\\\"cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wxxsq\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"opendatahub\"/\"odh-model-controller-7998c67dc4-kkjcw\": Internal error occurred: admission plugin \"ValidatingAdmissionWebhook\" failed to complete validation in 13s" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:40:56.869479 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:56.869445 2578 scope.go:117] "RemoveContainer" containerID="b428257107c2dc599015f76664541decf160b6e2f5a90c4f31f302b42e2ac281" Jul 06 19:40:56.869801 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:56.869787 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:40:56.898245 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:56.898218 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" Jul 06 19:40:56.898573 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:56.898561 2578 scope.go:117] "RemoveContainer" containerID="cc8e8756489398d4a62650cef6f4de6d384b86265df1f3151868d99e5d0d6115" Jul 06 19:40:57.480394 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:57.480361 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" event={"ID":"fc0a6e65-42be-43eb-8a55-363e166ecc16","Type":"ContainerStarted","Data":"80a227903b3c72357effa87efad30db39664e9595d2fefc4290adcfd1b5a7b25"} Jul 06 19:40:57.480846 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:57.480570 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" Jul 06 19:40:57.875608 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:57.875578 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" Jul 06 19:40:57.875972 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:57.875955 2578 scope.go:117] "RemoveContainer" containerID="af2d801c1a24c000d36d46b44b0cabdadfbc36698487048f9a199d03fdd8e6fa" Jul 06 19:40:58.487021 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:58.486988 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" event={"ID":"40393be6-bd01-4617-ace4-5cbdfcd186fd","Type":"ContainerStarted","Data":"dcb0b1cd4f97ef7d6d37bb805e66fc3debc3fa87bba61791736d41ec602125a3"} Jul 06 19:40:58.487424 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:58.487196 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:40:58.488911 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:58.488884 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" event={"ID":"18778df4-9ca2-41fe-a983-e69f2d564dd2","Type":"ContainerStarted","Data":"558a8af47a67d61967a768d6428f226e5fe92ad21d4cce7ac4243e1844bb197e"} Jul 06 19:40:58.870108 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:40:58.870079 2578 scope.go:117] "RemoveContainer" containerID="3b91808346c79bc006dafe07af2c6eaece5805d1c0bd80fce84107c2b3c90e81" Jul 06 19:41:00.501501 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:00.501469 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" event={"ID":"f55184b1-bd4b-4da5-9f7a-d6099ed678ab","Type":"ContainerStarted","Data":"f0f820fbdea2bf273eaefc8f4aee06b77ed4ba5c9b8a9f81268d806fc0577ee6"} Jul 06 19:41:00.501932 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:00.501705 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:41:00.869241 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:00.869211 2578 scope.go:117] "RemoveContainer" containerID="a20488db2c13f8bb374cd2ea8d7b3a1118af42c57fe3a25d6d293b8cdb9f8488" Jul 06 19:41:00.895136 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:00.895106 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:41:01.508118 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:01.508015 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" event={"ID":"85caf392-2962-4d96-8ae0-59dbf3366abe","Type":"ContainerStarted","Data":"948fe49b9d65c18588c118d38cb4465c72128f287ae2e598e13db4570c459a3e"} Jul 06 19:41:01.508118 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:01.508081 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:41:02.870248 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:02.870213 2578 scope.go:117] "RemoveContainer" containerID="eb2756f794802cef8dc0f896eec3babdc853529c6836b5a5dae2a3845c1a52d8" Jul 06 19:41:03.521038 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:03.521001 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" event={"ID":"7f55d1c1-4236-45e8-812a-68f2b9015ff6","Type":"ContainerStarted","Data":"b81f7d8a4c24a9792b29f8f90940543451f09f0a7e50644c6a1887c712a5adcd"} Jul 06 19:41:03.521228 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:03.521212 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" Jul 06 19:41:08.489825 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:08.489783 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" Jul 06 19:41:08.491351 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:08.491323 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" Jul 06 19:41:08.491728 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:08.491689 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" Jul 06 19:41:12.515221 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:12.515193 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:41:14.528330 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:14.528300 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" Jul 06 19:41:29.499278 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:29.499250 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:41:31.510108 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:31.510082 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:41:33.518284 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.518193 2578 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" podUID="f55184b1-bd4b-4da5-9f7a-d6099ed678ab" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.31:8081/healthz\": dial tcp 10.132.0.31:8081: connect: connection refused" Jul 06 19:41:33.682519 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.682482 2578 generic.go:356] "Generic (PLEG): container finished" podID="85caf392-2962-4d96-8ae0-59dbf3366abe" containerID="948fe49b9d65c18588c118d38cb4465c72128f287ae2e598e13db4570c459a3e" exitCode=1 Jul 06 19:41:33.682738 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.682552 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" event={"ID":"85caf392-2962-4d96-8ae0-59dbf3366abe","Type":"ContainerDied","Data":"948fe49b9d65c18588c118d38cb4465c72128f287ae2e598e13db4570c459a3e"} Jul 06 19:41:33.682738 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.682653 2578 scope.go:117] "RemoveContainer" containerID="a20488db2c13f8bb374cd2ea8d7b3a1118af42c57fe3a25d6d293b8cdb9f8488" Jul 06 19:41:33.683001 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.682979 2578 scope.go:117] "RemoveContainer" containerID="948fe49b9d65c18588c118d38cb4465c72128f287ae2e598e13db4570c459a3e" Jul 06 19:41:33.683249 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:33.683225 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=odh-model-controller-7998c67dc4-kkjcw_opendatahub(85caf392-2962-4d96-8ae0-59dbf3366abe)\"" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" podUID="85caf392-2962-4d96-8ae0-59dbf3366abe" Jul 06 19:41:33.684433 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.684411 2578 generic.go:356] "Generic (PLEG): container finished" podID="f55184b1-bd4b-4da5-9f7a-d6099ed678ab" containerID="f0f820fbdea2bf273eaefc8f4aee06b77ed4ba5c9b8a9f81268d806fc0577ee6" exitCode=1 Jul 06 19:41:33.684510 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.684481 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" event={"ID":"f55184b1-bd4b-4da5-9f7a-d6099ed678ab","Type":"ContainerDied","Data":"f0f820fbdea2bf273eaefc8f4aee06b77ed4ba5c9b8a9f81268d806fc0577ee6"} Jul 06 19:41:33.684814 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.684798 2578 scope.go:117] "RemoveContainer" containerID="f0f820fbdea2bf273eaefc8f4aee06b77ed4ba5c9b8a9f81268d806fc0577ee6" Jul 06 19:41:33.685005 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:33.684985 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-hrvh5_opendatahub(f55184b1-bd4b-4da5-9f7a-d6099ed678ab)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" podUID="f55184b1-bd4b-4da5-9f7a-d6099ed678ab" Jul 06 19:41:33.686119 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.686100 2578 generic.go:356] "Generic (PLEG): container finished" podID="40393be6-bd01-4617-ace4-5cbdfcd186fd" containerID="dcb0b1cd4f97ef7d6d37bb805e66fc3debc3fa87bba61791736d41ec602125a3" exitCode=1 Jul 06 19:41:33.686200 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.686159 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" event={"ID":"40393be6-bd01-4617-ace4-5cbdfcd186fd","Type":"ContainerDied","Data":"dcb0b1cd4f97ef7d6d37bb805e66fc3debc3fa87bba61791736d41ec602125a3"} Jul 06 19:41:33.686426 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.686410 2578 scope.go:117] "RemoveContainer" containerID="dcb0b1cd4f97ef7d6d37bb805e66fc3debc3fa87bba61791736d41ec602125a3" Jul 06 19:41:33.686624 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:33.686603 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pv5d5_opendatahub(40393be6-bd01-4617-ace4-5cbdfcd186fd)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" podUID="40393be6-bd01-4617-ace4-5cbdfcd186fd" Jul 06 19:41:33.688133 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.688115 2578 generic.go:356] "Generic (PLEG): container finished" podID="18778df4-9ca2-41fe-a983-e69f2d564dd2" containerID="558a8af47a67d61967a768d6428f226e5fe92ad21d4cce7ac4243e1844bb197e" exitCode=1 Jul 06 19:41:33.688227 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.688184 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" event={"ID":"18778df4-9ca2-41fe-a983-e69f2d564dd2","Type":"ContainerDied","Data":"558a8af47a67d61967a768d6428f226e5fe92ad21d4cce7ac4243e1844bb197e"} Jul 06 19:41:33.688446 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.688431 2578 scope.go:117] "RemoveContainer" containerID="558a8af47a67d61967a768d6428f226e5fe92ad21d4cce7ac4243e1844bb197e" Jul 06 19:41:33.688591 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:33.688573 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=dns-operator-controller-manager-864dd78c4c-6br2w_kuadrant-system(18778df4-9ca2-41fe-a983-e69f2d564dd2)\"" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" podUID="18778df4-9ca2-41fe-a983-e69f2d564dd2" Jul 06 19:41:33.690408 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.690392 2578 generic.go:356] "Generic (PLEG): container finished" podID="f1f2d36b-4239-45fa-a6d7-85c316bd426e" containerID="545853f6be91f627ce1bde6d1bced56d13f20b8295054db42729a507835c6066" exitCode=1 Jul 06 19:41:33.690493 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.690451 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" event={"ID":"f1f2d36b-4239-45fa-a6d7-85c316bd426e","Type":"ContainerDied","Data":"545853f6be91f627ce1bde6d1bced56d13f20b8295054db42729a507835c6066"} Jul 06 19:41:33.690808 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.690791 2578 scope.go:117] "RemoveContainer" containerID="545853f6be91f627ce1bde6d1bced56d13f20b8295054db42729a507835c6066" Jul 06 19:41:33.690993 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:33.690976 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=opendatahub-operator-controller-manager-7865995f8c-mwbc8_opendatahub(f1f2d36b-4239-45fa-a6d7-85c316bd426e)\"" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" podUID="f1f2d36b-4239-45fa-a6d7-85c316bd426e" Jul 06 19:41:33.692385 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.692366 2578 generic.go:356] "Generic (PLEG): container finished" podID="7f55d1c1-4236-45e8-812a-68f2b9015ff6" containerID="b81f7d8a4c24a9792b29f8f90940543451f09f0a7e50644c6a1887c712a5adcd" exitCode=1 Jul 06 19:41:33.692497 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.692430 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" event={"ID":"7f55d1c1-4236-45e8-812a-68f2b9015ff6","Type":"ContainerDied","Data":"b81f7d8a4c24a9792b29f8f90940543451f09f0a7e50644c6a1887c712a5adcd"} Jul 06 19:41:33.692773 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.692756 2578 scope.go:117] "RemoveContainer" containerID="b81f7d8a4c24a9792b29f8f90940543451f09f0a7e50644c6a1887c712a5adcd" Jul 06 19:41:33.692927 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:33.692906 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=authorino-operator-75cc8d5cb7-ndlnz_kuadrant-system(7f55d1c1-4236-45e8-812a-68f2b9015ff6)\"" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" podUID="7f55d1c1-4236-45e8-812a-68f2b9015ff6" Jul 06 19:41:33.694224 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.694206 2578 generic.go:356] "Generic (PLEG): container finished" podID="fc0a6e65-42be-43eb-8a55-363e166ecc16" containerID="80a227903b3c72357effa87efad30db39664e9595d2fefc4290adcfd1b5a7b25" exitCode=1 Jul 06 19:41:33.694311 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.694273 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" event={"ID":"fc0a6e65-42be-43eb-8a55-363e166ecc16","Type":"ContainerDied","Data":"80a227903b3c72357effa87efad30db39664e9595d2fefc4290adcfd1b5a7b25"} Jul 06 19:41:33.694533 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.694519 2578 scope.go:117] "RemoveContainer" containerID="80a227903b3c72357effa87efad30db39664e9595d2fefc4290adcfd1b5a7b25" Jul 06 19:41:33.694664 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:33.694647 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=limitador-operator-controller-manager-859d7b78bd-4lrbz_kuadrant-system(fc0a6e65-42be-43eb-8a55-363e166ecc16)\"" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" podUID="fc0a6e65-42be-43eb-8a55-363e166ecc16" Jul 06 19:41:33.695453 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.695439 2578 scope.go:117] "RemoveContainer" containerID="3b91808346c79bc006dafe07af2c6eaece5805d1c0bd80fce84107c2b3c90e81" Jul 06 19:41:33.710320 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.710300 2578 scope.go:117] "RemoveContainer" containerID="b428257107c2dc599015f76664541decf160b6e2f5a90c4f31f302b42e2ac281" Jul 06 19:41:33.722570 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.722546 2578 scope.go:117] "RemoveContainer" containerID="af2d801c1a24c000d36d46b44b0cabdadfbc36698487048f9a199d03fdd8e6fa" Jul 06 19:41:33.734436 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.734416 2578 scope.go:117] "RemoveContainer" containerID="8256b2550654e5c34e69b16380fdb1c7d2a644b9b56f133f6c13b9738c4f339b" Jul 06 19:41:33.746187 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.746168 2578 scope.go:117] "RemoveContainer" containerID="eb2756f794802cef8dc0f896eec3babdc853529c6836b5a5dae2a3845c1a52d8" Jul 06 19:41:33.757881 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:33.757860 2578 scope.go:117] "RemoveContainer" containerID="cc8e8756489398d4a62650cef6f4de6d384b86265df1f3151868d99e5d0d6115" Jul 06 19:41:33.988087 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:33.988045 2578 controller.go:195] "Failed to update lease" err="Put \"https://a7094b167326a4c82a83b6b901f5856c-474eb07a5e9b8c2e.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-140-94.ec2.internal?timeout=10s\": context deadline exceeded" Jul 06 19:41:34.494821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:34.494788 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:41:34.527519 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:34.527490 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" Jul 06 19:41:34.704143 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:34.704114 2578 scope.go:117] "RemoveContainer" containerID="dcb0b1cd4f97ef7d6d37bb805e66fc3debc3fa87bba61791736d41ec602125a3" Jul 06 19:41:34.704392 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:34.704366 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pv5d5_opendatahub(40393be6-bd01-4617-ace4-5cbdfcd186fd)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" podUID="40393be6-bd01-4617-ace4-5cbdfcd186fd" Jul 06 19:41:34.708478 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:34.708456 2578 scope.go:117] "RemoveContainer" containerID="b81f7d8a4c24a9792b29f8f90940543451f09f0a7e50644c6a1887c712a5adcd" Jul 06 19:41:34.708665 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:34.708643 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=authorino-operator-75cc8d5cb7-ndlnz_kuadrant-system(7f55d1c1-4236-45e8-812a-68f2b9015ff6)\"" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" podUID="7f55d1c1-4236-45e8-812a-68f2b9015ff6" Jul 06 19:41:36.508445 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:36.508410 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:41:36.508867 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:36.508769 2578 scope.go:117] "RemoveContainer" containerID="f0f820fbdea2bf273eaefc8f4aee06b77ed4ba5c9b8a9f81268d806fc0577ee6" Jul 06 19:41:36.508981 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:36.508962 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-hrvh5_opendatahub(f55184b1-bd4b-4da5-9f7a-d6099ed678ab)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" podUID="f55184b1-bd4b-4da5-9f7a-d6099ed678ab" Jul 06 19:41:36.871039 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:36.871007 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:41:36.871376 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:36.871364 2578 scope.go:117] "RemoveContainer" containerID="545853f6be91f627ce1bde6d1bced56d13f20b8295054db42729a507835c6066" Jul 06 19:41:36.871562 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:36.871545 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=opendatahub-operator-controller-manager-7865995f8c-mwbc8_opendatahub(f1f2d36b-4239-45fa-a6d7-85c316bd426e)\"" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" podUID="f1f2d36b-4239-45fa-a6d7-85c316bd426e" Jul 06 19:41:36.898964 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:36.898937 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" Jul 06 19:41:36.899243 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:36.899231 2578 scope.go:117] "RemoveContainer" containerID="80a227903b3c72357effa87efad30db39664e9595d2fefc4290adcfd1b5a7b25" Jul 06 19:41:36.899417 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:36.899401 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=limitador-operator-controller-manager-859d7b78bd-4lrbz_kuadrant-system(fc0a6e65-42be-43eb-8a55-363e166ecc16)\"" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" podUID="fc0a6e65-42be-43eb-8a55-363e166ecc16" Jul 06 19:41:37.875554 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:37.875525 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" Jul 06 19:41:37.875956 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:37.875845 2578 scope.go:117] "RemoveContainer" containerID="558a8af47a67d61967a768d6428f226e5fe92ad21d4cce7ac4243e1844bb197e" Jul 06 19:41:37.876025 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:37.876008 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=dns-operator-controller-manager-864dd78c4c-6br2w_kuadrant-system(18778df4-9ca2-41fe-a983-e69f2d564dd2)\"" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" podUID="18778df4-9ca2-41fe-a983-e69f2d564dd2" Jul 06 19:41:38.489236 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:38.489205 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" Jul 06 19:41:38.489564 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:38.489552 2578 scope.go:117] "RemoveContainer" containerID="80a227903b3c72357effa87efad30db39664e9595d2fefc4290adcfd1b5a7b25" Jul 06 19:41:38.489784 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:38.489764 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=limitador-operator-controller-manager-859d7b78bd-4lrbz_kuadrant-system(fc0a6e65-42be-43eb-8a55-363e166ecc16)\"" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" podUID="fc0a6e65-42be-43eb-8a55-363e166ecc16" Jul 06 19:41:38.491586 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:38.491558 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" Jul 06 19:41:38.730181 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:38.730156 2578 scope.go:117] "RemoveContainer" containerID="558a8af47a67d61967a768d6428f226e5fe92ad21d4cce7ac4243e1844bb197e" Jul 06 19:41:38.730371 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:38.730352 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=dns-operator-controller-manager-864dd78c4c-6br2w_kuadrant-system(18778df4-9ca2-41fe-a983-e69f2d564dd2)\"" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" podUID="18778df4-9ca2-41fe-a983-e69f2d564dd2" Jul 06 19:41:39.115843 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:39.115785 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:41:39.116405 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:39.116385 2578 scope.go:117] "RemoveContainer" containerID="545853f6be91f627ce1bde6d1bced56d13f20b8295054db42729a507835c6066" Jul 06 19:41:39.116625 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:39.116603 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=opendatahub-operator-controller-manager-7865995f8c-mwbc8_opendatahub(f1f2d36b-4239-45fa-a6d7-85c316bd426e)\"" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" podUID="f1f2d36b-4239-45fa-a6d7-85c316bd426e" Jul 06 19:41:40.894504 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:40.894466 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:41:40.894955 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:40.894903 2578 scope.go:117] "RemoveContainer" containerID="948fe49b9d65c18588c118d38cb4465c72128f287ae2e598e13db4570c459a3e" Jul 06 19:41:40.895086 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:40.895068 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=odh-model-controller-7998c67dc4-kkjcw_opendatahub(85caf392-2962-4d96-8ae0-59dbf3366abe)\"" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" podUID="85caf392-2962-4d96-8ae0-59dbf3366abe" Jul 06 19:41:42.513564 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:42.513529 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:41:42.513996 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:42.513911 2578 scope.go:117] "RemoveContainer" containerID="948fe49b9d65c18588c118d38cb4465c72128f287ae2e598e13db4570c459a3e" Jul 06 19:41:42.514086 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:42.514067 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=odh-model-controller-7998c67dc4-kkjcw_opendatahub(85caf392-2962-4d96-8ae0-59dbf3366abe)\"" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" podUID="85caf392-2962-4d96-8ae0-59dbf3366abe" Jul 06 19:41:42.864655 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:42.864617 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:41:42.865031 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:42.865016 2578 scope.go:117] "RemoveContainer" containerID="dcb0b1cd4f97ef7d6d37bb805e66fc3debc3fa87bba61791736d41ec602125a3" Jul 06 19:41:42.865222 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:42.865204 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pv5d5_opendatahub(40393be6-bd01-4617-ace4-5cbdfcd186fd)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" podUID="40393be6-bd01-4617-ace4-5cbdfcd186fd" Jul 06 19:41:43.517860 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:43.517816 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:41:43.518254 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:43.518235 2578 scope.go:117] "RemoveContainer" containerID="f0f820fbdea2bf273eaefc8f4aee06b77ed4ba5c9b8a9f81268d806fc0577ee6" Jul 06 19:41:43.518419 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:43.518399 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-hrvh5_opendatahub(f55184b1-bd4b-4da5-9f7a-d6099ed678ab)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" podUID="f55184b1-bd4b-4da5-9f7a-d6099ed678ab" Jul 06 19:41:46.869451 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:46.869416 2578 scope.go:117] "RemoveContainer" containerID="b81f7d8a4c24a9792b29f8f90940543451f09f0a7e50644c6a1887c712a5adcd" Jul 06 19:41:46.869861 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:46.869625 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=authorino-operator-75cc8d5cb7-ndlnz_kuadrant-system(7f55d1c1-4236-45e8-812a-68f2b9015ff6)\"" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" podUID="7f55d1c1-4236-45e8-812a-68f2b9015ff6" Jul 06 19:41:49.178095 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:49.178059 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" Jul 06 19:41:49.178468 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:49.178451 2578 scope.go:117] "RemoveContainer" containerID="b81f7d8a4c24a9792b29f8f90940543451f09f0a7e50644c6a1887c712a5adcd" Jul 06 19:41:49.178646 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:49.178626 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=authorino-operator-75cc8d5cb7-ndlnz_kuadrant-system(7f55d1c1-4236-45e8-812a-68f2b9015ff6)\"" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" podUID="7f55d1c1-4236-45e8-812a-68f2b9015ff6" Jul 06 19:41:50.869835 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:50.869803 2578 scope.go:117] "RemoveContainer" containerID="558a8af47a67d61967a768d6428f226e5fe92ad21d4cce7ac4243e1844bb197e" Jul 06 19:41:50.870216 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:50.870014 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=dns-operator-controller-manager-864dd78c4c-6br2w_kuadrant-system(18778df4-9ca2-41fe-a983-e69f2d564dd2)\"" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" podUID="18778df4-9ca2-41fe-a983-e69f2d564dd2" Jul 06 19:41:52.869988 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:52.869955 2578 scope.go:117] "RemoveContainer" containerID="545853f6be91f627ce1bde6d1bced56d13f20b8295054db42729a507835c6066" Jul 06 19:41:53.806171 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:53.806136 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" event={"ID":"f1f2d36b-4239-45fa-a6d7-85c316bd426e","Type":"ContainerStarted","Data":"e0ebc746439b755c38d59475f5fc0efaa8667b151a86ac67931e7328b9beb931"} Jul 06 19:41:53.806492 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:53.806473 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:41:53.870222 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:53.870189 2578 scope.go:117] "RemoveContainer" containerID="80a227903b3c72357effa87efad30db39664e9595d2fefc4290adcfd1b5a7b25" Jul 06 19:41:53.870639 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:53.870266 2578 scope.go:117] "RemoveContainer" containerID="948fe49b9d65c18588c118d38cb4465c72128f287ae2e598e13db4570c459a3e" Jul 06 19:41:53.870639 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:41:53.870459 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=odh-model-controller-7998c67dc4-kkjcw_opendatahub(85caf392-2962-4d96-8ae0-59dbf3366abe)\"" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" podUID="85caf392-2962-4d96-8ae0-59dbf3366abe" Jul 06 19:41:54.813600 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:54.813564 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" event={"ID":"fc0a6e65-42be-43eb-8a55-363e166ecc16","Type":"ContainerStarted","Data":"daa5fce7ddcee6f82d6411122180aefbcfa85ef22a83ba986daa952fbf713f18"} Jul 06 19:41:54.813969 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:54.813947 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" Jul 06 19:41:56.869889 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:56.869806 2578 scope.go:117] "RemoveContainer" containerID="f0f820fbdea2bf273eaefc8f4aee06b77ed4ba5c9b8a9f81268d806fc0577ee6" Jul 06 19:41:57.829888 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:57.829853 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" event={"ID":"f55184b1-bd4b-4da5-9f7a-d6099ed678ab","Type":"ContainerStarted","Data":"a836e5643a5b01cb2c815639f2fb68e0a7e4d56072a2f9d84625087b08f5739d"} Jul 06 19:41:57.830078 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:57.830060 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:41:57.869590 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:57.869553 2578 scope.go:117] "RemoveContainer" containerID="dcb0b1cd4f97ef7d6d37bb805e66fc3debc3fa87bba61791736d41ec602125a3" Jul 06 19:41:58.836911 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:58.836877 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" event={"ID":"40393be6-bd01-4617-ace4-5cbdfcd186fd","Type":"ContainerStarted","Data":"3bc78f571a4133779d672dc91442a1ca3a2270d360fff69e9593f438937d3d44"} Jul 06 19:41:59.869554 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:41:59.869518 2578 scope.go:117] "RemoveContainer" containerID="b81f7d8a4c24a9792b29f8f90940543451f09f0a7e50644c6a1887c712a5adcd" Jul 06 19:42:00.848220 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:42:00.848186 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" event={"ID":"7f55d1c1-4236-45e8-812a-68f2b9015ff6","Type":"ContainerStarted","Data":"3dccf6b62a0f2bb1ead68edeb118935af145de57f9cf19c9ddc1758b2bef8e69"} Jul 06 19:42:00.848479 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:42:00.848393 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" Jul 06 19:42:03.837895 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:42:03.837855 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:42:04.815794 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:42:04.815765 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7865995f8c-mwbc8" Jul 06 19:42:05.820792 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:42:05.820761 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4lrbz" Jul 06 19:42:05.872345 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:42:05.872319 2578 scope.go:117] "RemoveContainer" containerID="558a8af47a67d61967a768d6428f226e5fe92ad21d4cce7ac4243e1844bb197e" Jul 06 19:42:05.872515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:42:05.872440 2578 scope.go:117] "RemoveContainer" containerID="948fe49b9d65c18588c118d38cb4465c72128f287ae2e598e13db4570c459a3e" Jul 06 19:42:05.872634 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:42:05.872610 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=odh-model-controller-7998c67dc4-kkjcw_opendatahub(85caf392-2962-4d96-8ae0-59dbf3366abe)\"" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" podUID="85caf392-2962-4d96-8ae0-59dbf3366abe" Jul 06 19:42:06.884920 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:42:06.884888 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" event={"ID":"18778df4-9ca2-41fe-a983-e69f2d564dd2","Type":"ContainerStarted","Data":"218d9d44b76a002cd3ae4840e61c339eea2ddc734e03b6e6a9ac116c4a47c96c"} Jul 06 19:42:06.885325 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:42:06.885199 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" Jul 06 19:42:11.857408 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:42:11.857375 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndlnz" Jul 06 19:42:17.869388 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:42:17.869354 2578 scope.go:117] "RemoveContainer" containerID="948fe49b9d65c18588c118d38cb4465c72128f287ae2e598e13db4570c459a3e" Jul 06 19:42:17.891580 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:42:17.891557 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-6br2w" Jul 06 19:42:18.949431 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:42:18.949399 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" event={"ID":"85caf392-2962-4d96-8ae0-59dbf3366abe","Type":"ContainerStarted","Data":"c0b8a5135e5efd3970d3481742b6a0c97a57377f5e5ef3c310b506fcd39de901"} Jul 06 19:42:18.949828 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:42:18.949618 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:42:28.839476 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:42:28.839443 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hrvh5" Jul 06 19:42:28.843159 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:42:28.843134 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pv5d5" Jul 06 19:42:29.956625 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:42:29.956595 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-kkjcw" Jul 06 19:42:35.890437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:42:35.890409 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 19:42:35.890975 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:42:35.890888 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 19:45:00.180593 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:00.180508 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-sbhn6"] Jul 06 19:45:00.181018 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:00.180990 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eb86db9-5e48-402a-b69d-f807ee491609" containerName="authorino" Jul 06 19:45:00.181018 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:00.181004 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eb86db9-5e48-402a-b69d-f807ee491609" containerName="authorino" Jul 06 19:45:00.181090 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:00.181069 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eb86db9-5e48-402a-b69d-f807ee491609" containerName="authorino" Jul 06 19:45:00.184151 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:00.184133 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-sbhn6" Jul 06 19:45:00.186585 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:00.186567 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-vwbwq\"" Jul 06 19:45:00.202061 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:00.202032 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-sbhn6"] Jul 06 19:45:00.318910 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:00.318877 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgs4f\" (UniqueName: \"kubernetes.io/projected/32fd3226-5ae9-4c8a-b8a7-6fec412df49c-kube-api-access-pgs4f\") pod \"maas-api-key-cleanup-29722785-sbhn6\" (UID: \"32fd3226-5ae9-4c8a-b8a7-6fec412df49c\") " pod="opendatahub/maas-api-key-cleanup-29722785-sbhn6" Jul 06 19:45:00.419652 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:00.419620 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgs4f\" (UniqueName: \"kubernetes.io/projected/32fd3226-5ae9-4c8a-b8a7-6fec412df49c-kube-api-access-pgs4f\") pod \"maas-api-key-cleanup-29722785-sbhn6\" (UID: \"32fd3226-5ae9-4c8a-b8a7-6fec412df49c\") " pod="opendatahub/maas-api-key-cleanup-29722785-sbhn6" Jul 06 19:45:00.429158 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:00.429125 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgs4f\" (UniqueName: \"kubernetes.io/projected/32fd3226-5ae9-4c8a-b8a7-6fec412df49c-kube-api-access-pgs4f\") pod \"maas-api-key-cleanup-29722785-sbhn6\" (UID: \"32fd3226-5ae9-4c8a-b8a7-6fec412df49c\") " pod="opendatahub/maas-api-key-cleanup-29722785-sbhn6" Jul 06 19:45:00.494242 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:00.494163 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-sbhn6" Jul 06 19:45:00.633294 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:00.633264 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-sbhn6"] Jul 06 19:45:00.635533 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:45:00.635503 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32fd3226_5ae9_4c8a_b8a7_6fec412df49c.slice/crio-b762c568b600e4cd430d837e4ef9a95d5f89b93ccdf485dea3e7edef28455f30 WatchSource:0}: Error finding container b762c568b600e4cd430d837e4ef9a95d5f89b93ccdf485dea3e7edef28455f30: Status 404 returned error can't find the container with id b762c568b600e4cd430d837e4ef9a95d5f89b93ccdf485dea3e7edef28455f30 Jul 06 19:45:00.777243 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:00.777204 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-sbhn6" event={"ID":"32fd3226-5ae9-4c8a-b8a7-6fec412df49c","Type":"ContainerStarted","Data":"b762c568b600e4cd430d837e4ef9a95d5f89b93ccdf485dea3e7edef28455f30"} Jul 06 19:45:03.797265 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:03.797236 2578 generic.go:356] "Generic (PLEG): container finished" podID="32fd3226-5ae9-4c8a-b8a7-6fec412df49c" containerID="2839d2fcb7af2b71751eb9fff3a87ae031f98760bc57c06efae47dbe38edb20a" exitCode=0 Jul 06 19:45:03.797680 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:03.797325 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-sbhn6" event={"ID":"32fd3226-5ae9-4c8a-b8a7-6fec412df49c","Type":"ContainerDied","Data":"2839d2fcb7af2b71751eb9fff3a87ae031f98760bc57c06efae47dbe38edb20a"} Jul 06 19:45:04.944743 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:04.944716 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-sbhn6" Jul 06 19:45:05.066925 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:05.066808 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgs4f\" (UniqueName: \"kubernetes.io/projected/32fd3226-5ae9-4c8a-b8a7-6fec412df49c-kube-api-access-pgs4f\") pod \"32fd3226-5ae9-4c8a-b8a7-6fec412df49c\" (UID: \"32fd3226-5ae9-4c8a-b8a7-6fec412df49c\") " Jul 06 19:45:05.069134 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:05.069107 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32fd3226-5ae9-4c8a-b8a7-6fec412df49c-kube-api-access-pgs4f" (OuterVolumeSpecName: "kube-api-access-pgs4f") pod "32fd3226-5ae9-4c8a-b8a7-6fec412df49c" (UID: "32fd3226-5ae9-4c8a-b8a7-6fec412df49c"). InnerVolumeSpecName "kube-api-access-pgs4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:45:05.168332 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:05.168279 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgs4f\" (UniqueName: \"kubernetes.io/projected/32fd3226-5ae9-4c8a-b8a7-6fec412df49c-kube-api-access-pgs4f\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:45:05.809777 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:05.809747 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-sbhn6" Jul 06 19:45:05.809777 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:05.809767 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-sbhn6" event={"ID":"32fd3226-5ae9-4c8a-b8a7-6fec412df49c","Type":"ContainerDied","Data":"b762c568b600e4cd430d837e4ef9a95d5f89b93ccdf485dea3e7edef28455f30"} Jul 06 19:45:05.809998 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:45:05.809838 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b762c568b600e4cd430d837e4ef9a95d5f89b93ccdf485dea3e7edef28455f30" Jul 06 19:47:35.931218 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:47:35.931186 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 19:47:35.933584 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:47:35.933562 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 19:52:35.964819 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:52:35.964794 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 19:52:35.970851 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:52:35.970828 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 19:53:38.156626 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.156593 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7"] Jul 06 19:53:38.157046 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.157032 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32fd3226-5ae9-4c8a-b8a7-6fec412df49c" containerName="cleanup" Jul 06 19:53:38.157091 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.157047 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="32fd3226-5ae9-4c8a-b8a7-6fec412df49c" containerName="cleanup" Jul 06 19:53:38.157140 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.157131 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="32fd3226-5ae9-4c8a-b8a7-6fec412df49c" containerName="cleanup" Jul 06 19:53:38.160127 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.160109 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.162771 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.162750 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-138f1c39-openshift-default-dockercfg-2dmkn\"" Jul 06 19:53:38.181393 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.181354 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7"] Jul 06 19:53:38.184747 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.184715 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-workload-certs\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.184877 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.184764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-data\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.184877 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.184788 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-workload-socket\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.184877 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.184803 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istiod-ca-cert\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.184988 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.184927 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t72j\" (UniqueName: \"kubernetes.io/projected/cb372c73-cc65-4f61-ae0e-d08511eaa29c-kube-api-access-8t72j\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.184988 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.184972 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-credential-socket\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.185082 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.185066 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-envoy\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.185126 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.185102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-podinfo\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.185197 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.185177 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-token\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.285867 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.285828 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t72j\" (UniqueName: \"kubernetes.io/projected/cb372c73-cc65-4f61-ae0e-d08511eaa29c-kube-api-access-8t72j\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.285867 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.285884 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-credential-socket\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.286135 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.285940 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-envoy\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.286135 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.285987 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-podinfo\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.286135 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.286021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-token\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.286135 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.286065 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-workload-certs\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.286135 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.286099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-data\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.286427 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.286135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-workload-socket\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.286427 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.286161 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istiod-ca-cert\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.286427 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.286397 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-credential-socket\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.286640 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.286618 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-data\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.286779 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.286756 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-workload-certs\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.286899 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.286856 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-workload-socket\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.287046 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.287026 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istiod-ca-cert\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.288423 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.288401 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-envoy\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.288546 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.288528 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-podinfo\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.296126 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.296103 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-token\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.296323 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.296306 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t72j\" (UniqueName: \"kubernetes.io/projected/cb372c73-cc65-4f61-ae0e-d08511eaa29c-kube-api-access-8t72j\") pod \"e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.472998 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.472905 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:38.627995 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.627963 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7"] Jul 06 19:53:38.630146 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:53:38.630114 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb372c73_cc65_4f61_ae0e_d08511eaa29c.slice/crio-0a67884c9ebd4219921f559d8ce7cdd5fdbaca5b29058173a173d2e668f860fe WatchSource:0}: Error finding container 0a67884c9ebd4219921f559d8ce7cdd5fdbaca5b29058173a173d2e668f860fe: Status 404 returned error can't find the container with id 0a67884c9ebd4219921f559d8ce7cdd5fdbaca5b29058173a173d2e668f860fe Jul 06 19:53:38.632326 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.632295 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:53:38.632400 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.632361 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:53:38.632400 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:38.632390 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:53:39.499282 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:39.499245 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" event={"ID":"cb372c73-cc65-4f61-ae0e-d08511eaa29c","Type":"ContainerStarted","Data":"e0796c28ea23bbd62620820ffce49ac9c3d03095dc44f85488de61aa4105a1b9"} Jul 06 19:53:39.499762 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:39.499301 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" event={"ID":"cb372c73-cc65-4f61-ae0e-d08511eaa29c","Type":"ContainerStarted","Data":"0a67884c9ebd4219921f559d8ce7cdd5fdbaca5b29058173a173d2e668f860fe"} Jul 06 19:53:39.531512 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:39.531446 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" podStartSLOduration=1.531424055 podStartE2EDuration="1.531424055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:53:39.529886111 +0000 UTC m=+1864.158840089" watchObservedRunningTime="2026-07-06 19:53:39.531424055 +0000 UTC m=+1864.160378024" Jul 06 19:53:40.473729 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:40.473660 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:40.478539 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:40.478514 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:40.504447 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:40.504412 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:40.505537 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:40.505518 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:44.521021 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:44.520984 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7"] Jul 06 19:53:44.521507 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:44.521179 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" podUID="cb372c73-cc65-4f61-ae0e-d08511eaa29c" containerName="istio-proxy" containerID="cri-o://e0796c28ea23bbd62620820ffce49ac9c3d03095dc44f85488de61aa4105a1b9" gracePeriod=30 Jul 06 19:53:50.617375 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.617340 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9"] Jul 06 19:53:50.623335 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.623303 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.625848 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.625818 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d34b5c74-openshift-default-dockercfg-s8rfl\"" Jul 06 19:53:50.648161 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.642616 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9"] Jul 06 19:53:50.705385 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.705344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s8tg\" (UniqueName: \"kubernetes.io/projected/c80a1205-8864-48fe-80ea-1757fde1c200-kube-api-access-2s8tg\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.705558 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.705454 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-credential-socket\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.705558 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.705505 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-workload-certs\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.705669 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.705616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80a1205-8864-48fe-80ea-1757fde1c200-istio-podinfo\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.705772 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.705678 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-istio-envoy\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.705843 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.705760 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80a1205-8864-48fe-80ea-1757fde1c200-istiod-ca-cert\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.706015 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.705994 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-istio-data\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.706150 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.706029 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80a1205-8864-48fe-80ea-1757fde1c200-istio-token\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.706150 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.706125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-workload-socket\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.807431 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.807391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80a1205-8864-48fe-80ea-1757fde1c200-istio-podinfo\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.807431 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.807433 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-istio-envoy\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.807679 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.807455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80a1205-8864-48fe-80ea-1757fde1c200-istiod-ca-cert\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.807679 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.807475 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-istio-data\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.807679 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.807492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80a1205-8864-48fe-80ea-1757fde1c200-istio-token\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.807679 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.807530 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-workload-socket\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.807679 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.807574 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s8tg\" (UniqueName: \"kubernetes.io/projected/c80a1205-8864-48fe-80ea-1757fde1c200-kube-api-access-2s8tg\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.807679 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.807637 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-credential-socket\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.807679 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.807676 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-workload-certs\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.808150 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.808028 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-workload-socket\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.808226 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.808195 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-workload-certs\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.808226 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.808207 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-istio-data\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.808437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.808409 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-credential-socket\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.808787 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.808762 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80a1205-8864-48fe-80ea-1757fde1c200-istiod-ca-cert\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.809892 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.809868 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-istio-envoy\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.810194 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.810171 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80a1205-8864-48fe-80ea-1757fde1c200-istio-podinfo\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.823512 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.823476 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s8tg\" (UniqueName: \"kubernetes.io/projected/c80a1205-8864-48fe-80ea-1757fde1c200-kube-api-access-2s8tg\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.824760 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.824735 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80a1205-8864-48fe-80ea-1757fde1c200-istio-token\") pod \"e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:50.937744 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:50.937632 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:51.103101 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:51.103068 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9"] Jul 06 19:53:51.105356 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:53:51.105321 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80a1205_8864_48fe_80ea_1757fde1c200.slice/crio-9e1abca21d290d2cc4b3fd310abb52533e58d16b20bd3d5e4278bd6abb9dd35c WatchSource:0}: Error finding container 9e1abca21d290d2cc4b3fd310abb52533e58d16b20bd3d5e4278bd6abb9dd35c: Status 404 returned error can't find the container with id 9e1abca21d290d2cc4b3fd310abb52533e58d16b20bd3d5e4278bd6abb9dd35c Jul 06 19:53:51.107604 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:51.107568 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:53:51.107820 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:51.107635 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:53:51.107820 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:51.107667 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:53:51.577604 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:51.577572 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" event={"ID":"c80a1205-8864-48fe-80ea-1757fde1c200","Type":"ContainerStarted","Data":"6aa17b9d59257ba306d786049445db90880717541463f583db202657d5d410bf"} Jul 06 19:53:51.577832 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:51.577626 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" event={"ID":"c80a1205-8864-48fe-80ea-1757fde1c200","Type":"ContainerStarted","Data":"9e1abca21d290d2cc4b3fd310abb52533e58d16b20bd3d5e4278bd6abb9dd35c"} Jul 06 19:53:51.597421 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:51.597367 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" podStartSLOduration=1.597345976 podStartE2EDuration="1.597345976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:53:51.596518535 +0000 UTC m=+1876.225472501" watchObservedRunningTime="2026-07-06 19:53:51.597345976 +0000 UTC m=+1876.226299944" Jul 06 19:53:51.897780 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:51.897755 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:51.938638 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:51.938610 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:52.020588 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.020550 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t72j\" (UniqueName: \"kubernetes.io/projected/cb372c73-cc65-4f61-ae0e-d08511eaa29c-kube-api-access-8t72j\") pod \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " Jul 06 19:53:52.020800 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.020603 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-workload-certs\") pod \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " Jul 06 19:53:52.020800 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.020640 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-envoy\") pod \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " Jul 06 19:53:52.020800 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.020754 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-token\") pod \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " Jul 06 19:53:52.020981 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.020959 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cb372c73-cc65-4f61-ae0e-d08511eaa29c" (UID: "cb372c73-cc65-4f61-ae0e-d08511eaa29c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:53:52.021186 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.021167 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-credential-socket\") pod \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " Jul 06 19:53:52.021263 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.021227 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-workload-socket\") pod \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " Jul 06 19:53:52.021322 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.021284 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-data\") pod \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " Jul 06 19:53:52.021322 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.021310 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istiod-ca-cert\") pod \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " Jul 06 19:53:52.021434 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.021338 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-podinfo\") pod \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\" (UID: \"cb372c73-cc65-4f61-ae0e-d08511eaa29c\") " Jul 06 19:53:52.021434 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.021393 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cb372c73-cc65-4f61-ae0e-d08511eaa29c" (UID: "cb372c73-cc65-4f61-ae0e-d08511eaa29c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:53:52.022053 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.021714 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cb372c73-cc65-4f61-ae0e-d08511eaa29c" (UID: "cb372c73-cc65-4f61-ae0e-d08511eaa29c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:53:52.022053 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.021730 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-workload-certs\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:53:52.022053 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.021749 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-credential-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:53:52.022053 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.022018 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cb372c73-cc65-4f61-ae0e-d08511eaa29c" (UID: "cb372c73-cc65-4f61-ae0e-d08511eaa29c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:53:52.022338 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.022163 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-data" (OuterVolumeSpecName: "istio-data") pod "cb372c73-cc65-4f61-ae0e-d08511eaa29c" (UID: "cb372c73-cc65-4f61-ae0e-d08511eaa29c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:53:52.024952 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.024893 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cb372c73-cc65-4f61-ae0e-d08511eaa29c" (UID: "cb372c73-cc65-4f61-ae0e-d08511eaa29c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:53:52.024952 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.024905 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-token" (OuterVolumeSpecName: "istio-token") pod "cb372c73-cc65-4f61-ae0e-d08511eaa29c" (UID: "cb372c73-cc65-4f61-ae0e-d08511eaa29c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:53:52.025391 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.025364 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cb372c73-cc65-4f61-ae0e-d08511eaa29c" (UID: "cb372c73-cc65-4f61-ae0e-d08511eaa29c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:53:52.025838 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.025805 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb372c73-cc65-4f61-ae0e-d08511eaa29c-kube-api-access-8t72j" (OuterVolumeSpecName: "kube-api-access-8t72j") pod "cb372c73-cc65-4f61-ae0e-d08511eaa29c" (UID: "cb372c73-cc65-4f61-ae0e-d08511eaa29c"). InnerVolumeSpecName "kube-api-access-8t72j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:53:52.122586 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.122548 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-token\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:53:52.122586 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.122580 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-workload-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:53:52.122586 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.122588 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-data\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:53:52.122863 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.122602 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istiod-ca-cert\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:53:52.122863 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.122612 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-podinfo\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:53:52.122863 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.122620 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t72j\" (UniqueName: \"kubernetes.io/projected/cb372c73-cc65-4f61-ae0e-d08511eaa29c-kube-api-access-8t72j\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:53:52.122863 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.122629 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb372c73-cc65-4f61-ae0e-d08511eaa29c-istio-envoy\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:53:52.584138 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.584107 2578 generic.go:356] "Generic (PLEG): container finished" podID="cb372c73-cc65-4f61-ae0e-d08511eaa29c" containerID="e0796c28ea23bbd62620820ffce49ac9c3d03095dc44f85488de61aa4105a1b9" exitCode=0 Jul 06 19:53:52.584304 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.584149 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" Jul 06 19:53:52.584304 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.584192 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" event={"ID":"cb372c73-cc65-4f61-ae0e-d08511eaa29c","Type":"ContainerDied","Data":"e0796c28ea23bbd62620820ffce49ac9c3d03095dc44f85488de61aa4105a1b9"} Jul 06 19:53:52.584304 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.584248 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7" event={"ID":"cb372c73-cc65-4f61-ae0e-d08511eaa29c","Type":"ContainerDied","Data":"0a67884c9ebd4219921f559d8ce7cdd5fdbaca5b29058173a173d2e668f860fe"} Jul 06 19:53:52.584304 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.584280 2578 scope.go:117] "RemoveContainer" containerID="e0796c28ea23bbd62620820ffce49ac9c3d03095dc44f85488de61aa4105a1b9" Jul 06 19:53:52.594275 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.594254 2578 scope.go:117] "RemoveContainer" containerID="e0796c28ea23bbd62620820ffce49ac9c3d03095dc44f85488de61aa4105a1b9" Jul 06 19:53:52.594506 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:53:52.594488 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0796c28ea23bbd62620820ffce49ac9c3d03095dc44f85488de61aa4105a1b9\": container with ID starting with e0796c28ea23bbd62620820ffce49ac9c3d03095dc44f85488de61aa4105a1b9 not found: ID does not exist" containerID="e0796c28ea23bbd62620820ffce49ac9c3d03095dc44f85488de61aa4105a1b9" Jul 06 19:53:52.594549 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.594516 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0796c28ea23bbd62620820ffce49ac9c3d03095dc44f85488de61aa4105a1b9"} err="failed to get container status \"e0796c28ea23bbd62620820ffce49ac9c3d03095dc44f85488de61aa4105a1b9\": rpc error: code = NotFound desc = could not find container \"e0796c28ea23bbd62620820ffce49ac9c3d03095dc44f85488de61aa4105a1b9\": container with ID starting with e0796c28ea23bbd62620820ffce49ac9c3d03095dc44f85488de61aa4105a1b9 not found: ID does not exist" Jul 06 19:53:52.606482 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.606451 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7"] Jul 06 19:53:52.610670 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.610648 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-138f1c39-openshift-default-57df879c7f-md4g7"] Jul 06 19:53:52.943556 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:52.943480 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:53.591238 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:53.591211 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:53.592327 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:53.592308 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:53:53.875552 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:53:53.875521 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb372c73-cc65-4f61-ae0e-d08511eaa29c" path="/var/lib/kubelet/pods/cb372c73-cc65-4f61-ae0e-d08511eaa29c/volumes" Jul 06 19:54:02.969941 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:02.969901 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9"] Jul 06 19:54:02.970361 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:02.970164 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" podUID="c80a1205-8864-48fe-80ea-1757fde1c200" containerName="istio-proxy" containerID="cri-o://6aa17b9d59257ba306d786049445db90880717541463f583db202657d5d410bf" gracePeriod=30 Jul 06 19:54:03.842246 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.842212 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:54:03.935467 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.935434 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-istio-data\") pod \"c80a1205-8864-48fe-80ea-1757fde1c200\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " Jul 06 19:54:03.935467 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.935466 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-workload-socket\") pod \"c80a1205-8864-48fe-80ea-1757fde1c200\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " Jul 06 19:54:03.935721 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.935490 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s8tg\" (UniqueName: \"kubernetes.io/projected/c80a1205-8864-48fe-80ea-1757fde1c200-kube-api-access-2s8tg\") pod \"c80a1205-8864-48fe-80ea-1757fde1c200\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " Jul 06 19:54:03.935721 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.935565 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-credential-socket\") pod \"c80a1205-8864-48fe-80ea-1757fde1c200\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " Jul 06 19:54:03.935721 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.935705 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80a1205-8864-48fe-80ea-1757fde1c200-istio-token\") pod \"c80a1205-8864-48fe-80ea-1757fde1c200\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " Jul 06 19:54:03.935897 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.935765 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80a1205-8864-48fe-80ea-1757fde1c200-istiod-ca-cert\") pod \"c80a1205-8864-48fe-80ea-1757fde1c200\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " Jul 06 19:54:03.935897 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.935787 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c80a1205-8864-48fe-80ea-1757fde1c200" (UID: "c80a1205-8864-48fe-80ea-1757fde1c200"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:03.935897 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.935813 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-istio-envoy\") pod \"c80a1205-8864-48fe-80ea-1757fde1c200\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " Jul 06 19:54:03.936052 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.935906 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80a1205-8864-48fe-80ea-1757fde1c200-istio-podinfo\") pod \"c80a1205-8864-48fe-80ea-1757fde1c200\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " Jul 06 19:54:03.936052 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.935964 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-workload-certs\") pod \"c80a1205-8864-48fe-80ea-1757fde1c200\" (UID: \"c80a1205-8864-48fe-80ea-1757fde1c200\") " Jul 06 19:54:03.936386 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.935906 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c80a1205-8864-48fe-80ea-1757fde1c200" (UID: "c80a1205-8864-48fe-80ea-1757fde1c200"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:03.936386 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.936206 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80a1205-8864-48fe-80ea-1757fde1c200-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c80a1205-8864-48fe-80ea-1757fde1c200" (UID: "c80a1205-8864-48fe-80ea-1757fde1c200"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:54:03.936386 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.936327 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-credential-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:54:03.936386 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.936348 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80a1205-8864-48fe-80ea-1757fde1c200-istiod-ca-cert\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:54:03.936386 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.936364 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-workload-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:54:03.936746 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.936398 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c80a1205-8864-48fe-80ea-1757fde1c200" (UID: "c80a1205-8864-48fe-80ea-1757fde1c200"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:03.936746 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.936478 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-istio-data" (OuterVolumeSpecName: "istio-data") pod "c80a1205-8864-48fe-80ea-1757fde1c200" (UID: "c80a1205-8864-48fe-80ea-1757fde1c200"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:03.937876 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.937847 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80a1205-8864-48fe-80ea-1757fde1c200-kube-api-access-2s8tg" (OuterVolumeSpecName: "kube-api-access-2s8tg") pod "c80a1205-8864-48fe-80ea-1757fde1c200" (UID: "c80a1205-8864-48fe-80ea-1757fde1c200"). InnerVolumeSpecName "kube-api-access-2s8tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:54:03.938378 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.938354 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80a1205-8864-48fe-80ea-1757fde1c200-istio-token" (OuterVolumeSpecName: "istio-token") pod "c80a1205-8864-48fe-80ea-1757fde1c200" (UID: "c80a1205-8864-48fe-80ea-1757fde1c200"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:54:03.938455 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.938372 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c80a1205-8864-48fe-80ea-1757fde1c200" (UID: "c80a1205-8864-48fe-80ea-1757fde1c200"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:03.938455 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:03.938378 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c80a1205-8864-48fe-80ea-1757fde1c200-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c80a1205-8864-48fe-80ea-1757fde1c200" (UID: "c80a1205-8864-48fe-80ea-1757fde1c200"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:54:04.037433 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:04.037400 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80a1205-8864-48fe-80ea-1757fde1c200-istio-token\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:54:04.037433 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:04.037430 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-istio-envoy\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:54:04.037433 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:04.037441 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80a1205-8864-48fe-80ea-1757fde1c200-istio-podinfo\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:54:04.037899 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:04.037454 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-workload-certs\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:54:04.037899 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:04.037465 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80a1205-8864-48fe-80ea-1757fde1c200-istio-data\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:54:04.037899 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:04.037476 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s8tg\" (UniqueName: \"kubernetes.io/projected/c80a1205-8864-48fe-80ea-1757fde1c200-kube-api-access-2s8tg\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:54:04.650529 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:04.650490 2578 generic.go:356] "Generic (PLEG): container finished" podID="c80a1205-8864-48fe-80ea-1757fde1c200" containerID="6aa17b9d59257ba306d786049445db90880717541463f583db202657d5d410bf" exitCode=0 Jul 06 19:54:04.650759 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:04.650546 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" event={"ID":"c80a1205-8864-48fe-80ea-1757fde1c200","Type":"ContainerDied","Data":"6aa17b9d59257ba306d786049445db90880717541463f583db202657d5d410bf"} Jul 06 19:54:04.650759 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:04.650562 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" Jul 06 19:54:04.650759 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:04.650614 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9" event={"ID":"c80a1205-8864-48fe-80ea-1757fde1c200","Type":"ContainerDied","Data":"9e1abca21d290d2cc4b3fd310abb52533e58d16b20bd3d5e4278bd6abb9dd35c"} Jul 06 19:54:04.650759 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:04.650637 2578 scope.go:117] "RemoveContainer" containerID="6aa17b9d59257ba306d786049445db90880717541463f583db202657d5d410bf" Jul 06 19:54:04.660852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:04.660827 2578 scope.go:117] "RemoveContainer" containerID="6aa17b9d59257ba306d786049445db90880717541463f583db202657d5d410bf" Jul 06 19:54:04.661130 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:54:04.661112 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa17b9d59257ba306d786049445db90880717541463f583db202657d5d410bf\": container with ID starting with 6aa17b9d59257ba306d786049445db90880717541463f583db202657d5d410bf not found: ID does not exist" containerID="6aa17b9d59257ba306d786049445db90880717541463f583db202657d5d410bf" Jul 06 19:54:04.661185 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:04.661141 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa17b9d59257ba306d786049445db90880717541463f583db202657d5d410bf"} err="failed to get container status \"6aa17b9d59257ba306d786049445db90880717541463f583db202657d5d410bf\": rpc error: code = NotFound desc = could not find container \"6aa17b9d59257ba306d786049445db90880717541463f583db202657d5d410bf\": container with ID starting with 6aa17b9d59257ba306d786049445db90880717541463f583db202657d5d410bf not found: ID does not exist" Jul 06 19:54:04.672592 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:04.672563 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9"] Jul 06 19:54:04.676136 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:04.676115 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d34b5c74-openshift-default-6758d4b579-4jvq9"] Jul 06 19:54:05.875807 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:05.875770 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c80a1205-8864-48fe-80ea-1757fde1c200" path="/var/lib/kubelet/pods/c80a1205-8864-48fe-80ea-1757fde1c200/volumes" Jul 06 19:54:09.495747 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.495712 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n"] Jul 06 19:54:09.496390 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.496370 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb372c73-cc65-4f61-ae0e-d08511eaa29c" containerName="istio-proxy" Jul 06 19:54:09.496440 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.496395 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb372c73-cc65-4f61-ae0e-d08511eaa29c" containerName="istio-proxy" Jul 06 19:54:09.496440 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.496410 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c80a1205-8864-48fe-80ea-1757fde1c200" containerName="istio-proxy" Jul 06 19:54:09.496440 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.496419 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c80a1205-8864-48fe-80ea-1757fde1c200" containerName="istio-proxy" Jul 06 19:54:09.496588 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.496514 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c80a1205-8864-48fe-80ea-1757fde1c200" containerName="istio-proxy" Jul 06 19:54:09.496588 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.496527 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb372c73-cc65-4f61-ae0e-d08511eaa29c" containerName="istio-proxy" Jul 06 19:54:09.498943 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.498922 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.501345 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.501324 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2e7906ed-openshift-default-dockercfg-qcks9\"" Jul 06 19:54:09.521202 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.521177 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n"] Jul 06 19:54:09.599732 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.599685 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-workload-socket\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.599918 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.599742 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e88fde-babb-41a8-8364-140755497748-istio-token\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.599918 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.599781 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6jcd\" (UniqueName: \"kubernetes.io/projected/e4e88fde-babb-41a8-8364-140755497748-kube-api-access-b6jcd\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.599918 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.599812 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e88fde-babb-41a8-8364-140755497748-istiod-ca-cert\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.599918 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.599872 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-istio-envoy\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.600078 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.599956 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-workload-certs\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.600078 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.599978 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-istio-data\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.600078 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.599995 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e88fde-babb-41a8-8364-140755497748-istio-podinfo\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.600078 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.600037 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-credential-socket\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.700870 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.700833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-credential-socket\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.700870 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.700876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-workload-socket\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.701141 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.700899 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e88fde-babb-41a8-8364-140755497748-istio-token\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.701141 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.700932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6jcd\" (UniqueName: \"kubernetes.io/projected/e4e88fde-babb-41a8-8364-140755497748-kube-api-access-b6jcd\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.701141 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.700954 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e88fde-babb-41a8-8364-140755497748-istiod-ca-cert\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.701141 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.700974 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-istio-envoy\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.701141 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.701012 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-workload-certs\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.701141 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.701035 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-istio-data\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.701141 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.701059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e88fde-babb-41a8-8364-140755497748-istio-podinfo\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.701507 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.701336 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-credential-socket\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.701507 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.701418 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-workload-certs\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.701507 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.701448 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-istio-data\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.701676 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.701621 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-workload-socket\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.701796 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.701774 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e88fde-babb-41a8-8364-140755497748-istiod-ca-cert\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.703330 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.703307 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-istio-envoy\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.703521 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.703503 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e88fde-babb-41a8-8364-140755497748-istio-podinfo\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.710855 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.710828 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e88fde-babb-41a8-8364-140755497748-istio-token\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.711191 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.711175 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6jcd\" (UniqueName: \"kubernetes.io/projected/e4e88fde-babb-41a8-8364-140755497748-kube-api-access-b6jcd\") pod \"e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.810061 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.810026 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:09.964893 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.964857 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n"] Jul 06 19:54:09.968742 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:54:09.968714 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e88fde_babb_41a8_8364_140755497748.slice/crio-cc0f8361721913eec6fd19ad3bbdc775cd02321e060ab6b58c19bfeba0adb5bc WatchSource:0}: Error finding container cc0f8361721913eec6fd19ad3bbdc775cd02321e060ab6b58c19bfeba0adb5bc: Status 404 returned error can't find the container with id cc0f8361721913eec6fd19ad3bbdc775cd02321e060ab6b58c19bfeba0adb5bc Jul 06 19:54:09.971026 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.970984 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:54:09.971096 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.971079 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:54:09.971146 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:09.971128 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:54:10.683434 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:10.683390 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" event={"ID":"e4e88fde-babb-41a8-8364-140755497748","Type":"ContainerStarted","Data":"97231cf76fde8c4a7d1fa460ca511e7686d8fb93f05b38d51ed2987d7990ce90"} Jul 06 19:54:10.683960 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:10.683467 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" event={"ID":"e4e88fde-babb-41a8-8364-140755497748","Type":"ContainerStarted","Data":"cc0f8361721913eec6fd19ad3bbdc775cd02321e060ab6b58c19bfeba0adb5bc"} Jul 06 19:54:10.719299 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:10.719238 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" podStartSLOduration=1.71921694 podStartE2EDuration="1.71921694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:54:10.718249733 +0000 UTC m=+1895.347203696" watchObservedRunningTime="2026-07-06 19:54:10.71921694 +0000 UTC m=+1895.348170905" Jul 06 19:54:10.810602 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:10.810563 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:11.815523 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:11.815483 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:12.693854 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:12.693825 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:12.694939 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:12.694918 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:15.259673 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:15.259638 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n"] Jul 06 19:54:15.710112 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:15.710075 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" podUID="e4e88fde-babb-41a8-8364-140755497748" containerName="istio-proxy" containerID="cri-o://97231cf76fde8c4a7d1fa460ca511e7686d8fb93f05b38d51ed2987d7990ce90" gracePeriod=30 Jul 06 19:54:20.962037 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:20.962008 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:21.002928 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.002841 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-workload-certs\") pod \"e4e88fde-babb-41a8-8364-140755497748\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " Jul 06 19:54:21.002928 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.002881 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-credential-socket\") pod \"e4e88fde-babb-41a8-8364-140755497748\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " Jul 06 19:54:21.002928 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.002904 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-istio-data\") pod \"e4e88fde-babb-41a8-8364-140755497748\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " Jul 06 19:54:21.002928 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.002927 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e88fde-babb-41a8-8364-140755497748-istio-podinfo\") pod \"e4e88fde-babb-41a8-8364-140755497748\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " Jul 06 19:54:21.003282 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.002958 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e88fde-babb-41a8-8364-140755497748-istiod-ca-cert\") pod \"e4e88fde-babb-41a8-8364-140755497748\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " Jul 06 19:54:21.003282 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.003043 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-workload-socket\") pod \"e4e88fde-babb-41a8-8364-140755497748\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " Jul 06 19:54:21.003282 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.003085 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e88fde-babb-41a8-8364-140755497748-istio-token\") pod \"e4e88fde-babb-41a8-8364-140755497748\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " Jul 06 19:54:21.003282 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.003136 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6jcd\" (UniqueName: \"kubernetes.io/projected/e4e88fde-babb-41a8-8364-140755497748-kube-api-access-b6jcd\") pod \"e4e88fde-babb-41a8-8364-140755497748\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " Jul 06 19:54:21.003282 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.003202 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e4e88fde-babb-41a8-8364-140755497748" (UID: "e4e88fde-babb-41a8-8364-140755497748"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:21.003553 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.003372 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e4e88fde-babb-41a8-8364-140755497748" (UID: "e4e88fde-babb-41a8-8364-140755497748"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:21.003553 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.003527 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-istio-envoy\") pod \"e4e88fde-babb-41a8-8364-140755497748\" (UID: \"e4e88fde-babb-41a8-8364-140755497748\") " Jul 06 19:54:21.003670 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.003618 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e88fde-babb-41a8-8364-140755497748-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e4e88fde-babb-41a8-8364-140755497748" (UID: "e4e88fde-babb-41a8-8364-140755497748"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:54:21.003860 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.003832 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e4e88fde-babb-41a8-8364-140755497748" (UID: "e4e88fde-babb-41a8-8364-140755497748"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:21.003952 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.003905 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e88fde-babb-41a8-8364-140755497748-istiod-ca-cert\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:54:21.003952 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.003930 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-istio-data" (OuterVolumeSpecName: "istio-data") pod "e4e88fde-babb-41a8-8364-140755497748" (UID: "e4e88fde-babb-41a8-8364-140755497748"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:21.004072 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.003955 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-workload-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:54:21.004072 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.003970 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-credential-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:54:21.005596 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.005553 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4e88fde-babb-41a8-8364-140755497748-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e4e88fde-babb-41a8-8364-140755497748" (UID: "e4e88fde-babb-41a8-8364-140755497748"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:54:21.005894 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.005861 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e88fde-babb-41a8-8364-140755497748-kube-api-access-b6jcd" (OuterVolumeSpecName: "kube-api-access-b6jcd") pod "e4e88fde-babb-41a8-8364-140755497748" (UID: "e4e88fde-babb-41a8-8364-140755497748"). InnerVolumeSpecName "kube-api-access-b6jcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:54:21.005977 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.005948 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e4e88fde-babb-41a8-8364-140755497748" (UID: "e4e88fde-babb-41a8-8364-140755497748"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:21.005977 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.005955 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e88fde-babb-41a8-8364-140755497748-istio-token" (OuterVolumeSpecName: "istio-token") pod "e4e88fde-babb-41a8-8364-140755497748" (UID: "e4e88fde-babb-41a8-8364-140755497748"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:54:21.104429 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.104397 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e88fde-babb-41a8-8364-140755497748-istio-token\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:54:21.104429 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.104431 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6jcd\" (UniqueName: \"kubernetes.io/projected/e4e88fde-babb-41a8-8364-140755497748-kube-api-access-b6jcd\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:54:21.104635 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.104445 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-istio-envoy\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:54:21.104635 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.104459 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-workload-certs\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:54:21.104635 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.104470 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e88fde-babb-41a8-8364-140755497748-istio-data\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:54:21.104635 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.104484 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e88fde-babb-41a8-8364-140755497748-istio-podinfo\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:54:21.743652 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.743615 2578 generic.go:356] "Generic (PLEG): container finished" podID="e4e88fde-babb-41a8-8364-140755497748" containerID="97231cf76fde8c4a7d1fa460ca511e7686d8fb93f05b38d51ed2987d7990ce90" exitCode=0 Jul 06 19:54:21.743869 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.743667 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" Jul 06 19:54:21.743869 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.743734 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" event={"ID":"e4e88fde-babb-41a8-8364-140755497748","Type":"ContainerDied","Data":"97231cf76fde8c4a7d1fa460ca511e7686d8fb93f05b38d51ed2987d7990ce90"} Jul 06 19:54:21.743869 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.743790 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n" event={"ID":"e4e88fde-babb-41a8-8364-140755497748","Type":"ContainerDied","Data":"cc0f8361721913eec6fd19ad3bbdc775cd02321e060ab6b58c19bfeba0adb5bc"} Jul 06 19:54:21.743869 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.743820 2578 scope.go:117] "RemoveContainer" containerID="97231cf76fde8c4a7d1fa460ca511e7686d8fb93f05b38d51ed2987d7990ce90" Jul 06 19:54:21.754340 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.754321 2578 scope.go:117] "RemoveContainer" containerID="97231cf76fde8c4a7d1fa460ca511e7686d8fb93f05b38d51ed2987d7990ce90" Jul 06 19:54:21.754623 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:54:21.754603 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97231cf76fde8c4a7d1fa460ca511e7686d8fb93f05b38d51ed2987d7990ce90\": container with ID starting with 97231cf76fde8c4a7d1fa460ca511e7686d8fb93f05b38d51ed2987d7990ce90 not found: ID does not exist" containerID="97231cf76fde8c4a7d1fa460ca511e7686d8fb93f05b38d51ed2987d7990ce90" Jul 06 19:54:21.754715 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.754637 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97231cf76fde8c4a7d1fa460ca511e7686d8fb93f05b38d51ed2987d7990ce90"} err="failed to get container status \"97231cf76fde8c4a7d1fa460ca511e7686d8fb93f05b38d51ed2987d7990ce90\": rpc error: code = NotFound desc = could not find container \"97231cf76fde8c4a7d1fa460ca511e7686d8fb93f05b38d51ed2987d7990ce90\": container with ID starting with 97231cf76fde8c4a7d1fa460ca511e7686d8fb93f05b38d51ed2987d7990ce90 not found: ID does not exist" Jul 06 19:54:21.783664 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.783630 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n"] Jul 06 19:54:21.790840 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.790811 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2e7906ed-openshift-default-c7fbb98f5-bwr5n"] Jul 06 19:54:21.875955 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:54:21.875924 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4e88fde-babb-41a8-8364-140755497748" path="/var/lib/kubelet/pods/e4e88fde-babb-41a8-8364-140755497748/volumes" Jul 06 19:56:00.102859 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.102821 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq"] Jul 06 19:56:00.103347 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.103253 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4e88fde-babb-41a8-8364-140755497748" containerName="istio-proxy" Jul 06 19:56:00.103347 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.103269 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4e88fde-babb-41a8-8364-140755497748" containerName="istio-proxy" Jul 06 19:56:00.103421 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.103351 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4e88fde-babb-41a8-8364-140755497748" containerName="istio-proxy" Jul 06 19:56:00.106336 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.106318 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.108905 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.108882 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-03cdb5ed-openshift-default-dockercfg-j8b4l\"" Jul 06 19:56:00.132705 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.132651 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq"] Jul 06 19:56:00.285772 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.285735 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-istio-envoy\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.285772 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.285778 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-istio-data\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.286062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.285805 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc5c4c99-d284-4441-984d-b03d39642786-istiod-ca-cert\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.286062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.285922 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-workload-socket\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.286062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.285982 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc5c4c99-d284-4441-984d-b03d39642786-istio-podinfo\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.286062 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.286020 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc5c4c99-d284-4441-984d-b03d39642786-istio-token\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.286266 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.286078 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-workload-certs\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.286266 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.286128 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-credential-socket\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.286266 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.286169 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdmht\" (UniqueName: \"kubernetes.io/projected/bc5c4c99-d284-4441-984d-b03d39642786-kube-api-access-kdmht\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.386933 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.386833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-credential-socket\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.386933 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.386877 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdmht\" (UniqueName: \"kubernetes.io/projected/bc5c4c99-d284-4441-984d-b03d39642786-kube-api-access-kdmht\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.386933 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.386917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-istio-envoy\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.387267 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.386939 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-istio-data\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.387267 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.386959 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc5c4c99-d284-4441-984d-b03d39642786-istiod-ca-cert\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.387267 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.386987 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-workload-socket\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.387267 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.387021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc5c4c99-d284-4441-984d-b03d39642786-istio-podinfo\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.387267 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.387049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc5c4c99-d284-4441-984d-b03d39642786-istio-token\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.387267 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.387097 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-workload-certs\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.387601 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.387316 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-credential-socket\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.387601 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.387332 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-istio-data\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.387601 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.387414 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-workload-socket\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.387601 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.387594 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-workload-certs\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.387940 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.387917 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc5c4c99-d284-4441-984d-b03d39642786-istiod-ca-cert\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.389408 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.389375 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-istio-envoy\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.389582 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.389563 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc5c4c99-d284-4441-984d-b03d39642786-istio-podinfo\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.400295 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.400270 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdmht\" (UniqueName: \"kubernetes.io/projected/bc5c4c99-d284-4441-984d-b03d39642786-kube-api-access-kdmht\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.406984 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.406960 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc5c4c99-d284-4441-984d-b03d39642786-istio-token\") pod \"e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.419852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.419832 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:00.589237 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.589209 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq"] Jul 06 19:56:00.591393 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:56:00.591365 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5c4c99_d284_4441_984d_b03d39642786.slice/crio-438e03fa2052cbfdf53bcc2c272eaa197b4bf3fbffb31ae19248eef8cb50138e WatchSource:0}: Error finding container 438e03fa2052cbfdf53bcc2c272eaa197b4bf3fbffb31ae19248eef8cb50138e: Status 404 returned error can't find the container with id 438e03fa2052cbfdf53bcc2c272eaa197b4bf3fbffb31ae19248eef8cb50138e Jul 06 19:56:00.593427 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.593396 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:56:00.593487 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.593466 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:56:00.593524 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:00.593493 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:56:01.266542 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:01.266506 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" event={"ID":"bc5c4c99-d284-4441-984d-b03d39642786","Type":"ContainerStarted","Data":"55dd92bbd6385a591d52a22e1a34cf4ac490c08a90d3788c57665115e3bc40ba"} Jul 06 19:56:01.267036 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:01.266558 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" event={"ID":"bc5c4c99-d284-4441-984d-b03d39642786","Type":"ContainerStarted","Data":"438e03fa2052cbfdf53bcc2c272eaa197b4bf3fbffb31ae19248eef8cb50138e"} Jul 06 19:56:01.289019 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:01.288961 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" podStartSLOduration=1.288943236 podStartE2EDuration="1.288943236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:56:01.287977932 +0000 UTC m=+2005.916931900" watchObservedRunningTime="2026-07-06 19:56:01.288943236 +0000 UTC m=+2005.917897209" Jul 06 19:56:01.420046 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:01.420015 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:01.425078 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:01.425051 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:02.272175 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:02.272139 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:02.273135 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:02.273108 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:18.313881 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:18.313828 2578 patch_prober.go:28] interesting pod/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 19:56:18.316236 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:18.313906 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" podUID="bc5c4c99-d284-4441-984d-b03d39642786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 19:56:19.376968 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:19.376935 2578 generic.go:356] "Generic (PLEG): container finished" podID="bc5c4c99-d284-4441-984d-b03d39642786" containerID="55dd92bbd6385a591d52a22e1a34cf4ac490c08a90d3788c57665115e3bc40ba" exitCode=0 Jul 06 19:56:19.377377 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:19.377011 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" event={"ID":"bc5c4c99-d284-4441-984d-b03d39642786","Type":"ContainerDied","Data":"55dd92bbd6385a591d52a22e1a34cf4ac490c08a90d3788c57665115e3bc40ba"} Jul 06 19:56:19.377484 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:19.377467 2578 scope.go:117] "RemoveContainer" containerID="55dd92bbd6385a591d52a22e1a34cf4ac490c08a90d3788c57665115e3bc40ba" Jul 06 19:56:19.378508 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:19.378477 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:56:19.378588 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:19.378572 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:56:19.378672 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:19.378611 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:56:20.385259 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:20.385225 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" event={"ID":"bc5c4c99-d284-4441-984d-b03d39642786","Type":"ContainerStarted","Data":"0138bc7166a09e38dc1fbc9841cc5002475f037180f68c47da7ec7a7cb588eee"} Jul 06 19:56:20.420380 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:20.420346 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:20.423758 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:20.423736 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:21.390266 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:21.390236 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:21.391270 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:21.391247 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:22.630347 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:22.630313 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq"] Jul 06 19:56:23.400806 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:23.400774 2578 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" secret="" err="secret \"e2e-mt-03cdb5ed-openshift-default-dockercfg-j8b4l\" not found" Jul 06 19:56:24.406957 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:24.406895 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" podUID="bc5c4c99-d284-4441-984d-b03d39642786" containerName="istio-proxy" containerID="cri-o://0138bc7166a09e38dc1fbc9841cc5002475f037180f68c47da7ec7a7cb588eee" gracePeriod=30 Jul 06 19:56:29.666980 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.666916 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:29.751228 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.751189 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc5c4c99-d284-4441-984d-b03d39642786-istiod-ca-cert\") pod \"bc5c4c99-d284-4441-984d-b03d39642786\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " Jul 06 19:56:29.751228 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.751238 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdmht\" (UniqueName: \"kubernetes.io/projected/bc5c4c99-d284-4441-984d-b03d39642786-kube-api-access-kdmht\") pod \"bc5c4c99-d284-4441-984d-b03d39642786\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " Jul 06 19:56:29.751499 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.751269 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-istio-data\") pod \"bc5c4c99-d284-4441-984d-b03d39642786\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " Jul 06 19:56:29.751499 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.751303 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-workload-certs\") pod \"bc5c4c99-d284-4441-984d-b03d39642786\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " Jul 06 19:56:29.751499 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.751346 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc5c4c99-d284-4441-984d-b03d39642786-istio-podinfo\") pod \"bc5c4c99-d284-4441-984d-b03d39642786\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " Jul 06 19:56:29.751499 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.751366 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc5c4c99-d284-4441-984d-b03d39642786-istio-token\") pod \"bc5c4c99-d284-4441-984d-b03d39642786\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " Jul 06 19:56:29.751499 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.751415 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-istio-envoy\") pod \"bc5c4c99-d284-4441-984d-b03d39642786\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " Jul 06 19:56:29.751499 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.751446 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-workload-socket\") pod \"bc5c4c99-d284-4441-984d-b03d39642786\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " Jul 06 19:56:29.751499 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.751492 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-credential-socket\") pod \"bc5c4c99-d284-4441-984d-b03d39642786\" (UID: \"bc5c4c99-d284-4441-984d-b03d39642786\") " Jul 06 19:56:29.751900 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.751666 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bc5c4c99-d284-4441-984d-b03d39642786" (UID: "bc5c4c99-d284-4441-984d-b03d39642786"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:29.751900 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.751803 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc5c4c99-d284-4441-984d-b03d39642786-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bc5c4c99-d284-4441-984d-b03d39642786" (UID: "bc5c4c99-d284-4441-984d-b03d39642786"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:56:29.751900 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.751857 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-workload-certs\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:56:29.752041 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.751941 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bc5c4c99-d284-4441-984d-b03d39642786" (UID: "bc5c4c99-d284-4441-984d-b03d39642786"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:29.752095 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.752071 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bc5c4c99-d284-4441-984d-b03d39642786" (UID: "bc5c4c99-d284-4441-984d-b03d39642786"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:29.752369 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.752340 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-istio-data" (OuterVolumeSpecName: "istio-data") pod "bc5c4c99-d284-4441-984d-b03d39642786" (UID: "bc5c4c99-d284-4441-984d-b03d39642786"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:29.753732 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.753688 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bc5c4c99-d284-4441-984d-b03d39642786" (UID: "bc5c4c99-d284-4441-984d-b03d39642786"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:29.753817 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.753757 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc5c4c99-d284-4441-984d-b03d39642786-kube-api-access-kdmht" (OuterVolumeSpecName: "kube-api-access-kdmht") pod "bc5c4c99-d284-4441-984d-b03d39642786" (UID: "bc5c4c99-d284-4441-984d-b03d39642786"). InnerVolumeSpecName "kube-api-access-kdmht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:56:29.753983 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.753968 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc5c4c99-d284-4441-984d-b03d39642786-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bc5c4c99-d284-4441-984d-b03d39642786" (UID: "bc5c4c99-d284-4441-984d-b03d39642786"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:56:29.754032 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.754008 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc5c4c99-d284-4441-984d-b03d39642786-istio-token" (OuterVolumeSpecName: "istio-token") pod "bc5c4c99-d284-4441-984d-b03d39642786" (UID: "bc5c4c99-d284-4441-984d-b03d39642786"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:56:29.853347 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.853309 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc5c4c99-d284-4441-984d-b03d39642786-istiod-ca-cert\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:56:29.853347 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.853342 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdmht\" (UniqueName: \"kubernetes.io/projected/bc5c4c99-d284-4441-984d-b03d39642786-kube-api-access-kdmht\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:56:29.853347 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.853355 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-istio-data\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:56:29.853604 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.853367 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc5c4c99-d284-4441-984d-b03d39642786-istio-podinfo\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:56:29.853604 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.853378 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc5c4c99-d284-4441-984d-b03d39642786-istio-token\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:56:29.853604 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.853388 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-istio-envoy\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:56:29.853604 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.853399 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-workload-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:56:29.853604 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:29.853409 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5c4c99-d284-4441-984d-b03d39642786-credential-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:56:30.441156 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:30.441120 2578 generic.go:356] "Generic (PLEG): container finished" podID="bc5c4c99-d284-4441-984d-b03d39642786" containerID="0138bc7166a09e38dc1fbc9841cc5002475f037180f68c47da7ec7a7cb588eee" exitCode=0 Jul 06 19:56:30.441379 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:30.441174 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" Jul 06 19:56:30.441379 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:30.441205 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" event={"ID":"bc5c4c99-d284-4441-984d-b03d39642786","Type":"ContainerDied","Data":"0138bc7166a09e38dc1fbc9841cc5002475f037180f68c47da7ec7a7cb588eee"} Jul 06 19:56:30.441379 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:30.441261 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq" event={"ID":"bc5c4c99-d284-4441-984d-b03d39642786","Type":"ContainerDied","Data":"438e03fa2052cbfdf53bcc2c272eaa197b4bf3fbffb31ae19248eef8cb50138e"} Jul 06 19:56:30.441379 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:30.441283 2578 scope.go:117] "RemoveContainer" containerID="0138bc7166a09e38dc1fbc9841cc5002475f037180f68c47da7ec7a7cb588eee" Jul 06 19:56:30.450671 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:30.450651 2578 scope.go:117] "RemoveContainer" containerID="55dd92bbd6385a591d52a22e1a34cf4ac490c08a90d3788c57665115e3bc40ba" Jul 06 19:56:30.459437 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:30.459418 2578 scope.go:117] "RemoveContainer" containerID="0138bc7166a09e38dc1fbc9841cc5002475f037180f68c47da7ec7a7cb588eee" Jul 06 19:56:30.459797 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:56:30.459768 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0138bc7166a09e38dc1fbc9841cc5002475f037180f68c47da7ec7a7cb588eee\": container with ID starting with 0138bc7166a09e38dc1fbc9841cc5002475f037180f68c47da7ec7a7cb588eee not found: ID does not exist" containerID="0138bc7166a09e38dc1fbc9841cc5002475f037180f68c47da7ec7a7cb588eee" Jul 06 19:56:30.459857 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:30.459808 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0138bc7166a09e38dc1fbc9841cc5002475f037180f68c47da7ec7a7cb588eee"} err="failed to get container status \"0138bc7166a09e38dc1fbc9841cc5002475f037180f68c47da7ec7a7cb588eee\": rpc error: code = NotFound desc = could not find container \"0138bc7166a09e38dc1fbc9841cc5002475f037180f68c47da7ec7a7cb588eee\": container with ID starting with 0138bc7166a09e38dc1fbc9841cc5002475f037180f68c47da7ec7a7cb588eee not found: ID does not exist" Jul 06 19:56:30.459857 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:30.459827 2578 scope.go:117] "RemoveContainer" containerID="55dd92bbd6385a591d52a22e1a34cf4ac490c08a90d3788c57665115e3bc40ba" Jul 06 19:56:30.460028 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:30.460007 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq"] Jul 06 19:56:30.460087 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:56:30.460048 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55dd92bbd6385a591d52a22e1a34cf4ac490c08a90d3788c57665115e3bc40ba\": container with ID starting with 55dd92bbd6385a591d52a22e1a34cf4ac490c08a90d3788c57665115e3bc40ba not found: ID does not exist" containerID="55dd92bbd6385a591d52a22e1a34cf4ac490c08a90d3788c57665115e3bc40ba" Jul 06 19:56:30.460134 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:30.460079 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55dd92bbd6385a591d52a22e1a34cf4ac490c08a90d3788c57665115e3bc40ba"} err="failed to get container status \"55dd92bbd6385a591d52a22e1a34cf4ac490c08a90d3788c57665115e3bc40ba\": rpc error: code = NotFound desc = could not find container \"55dd92bbd6385a591d52a22e1a34cf4ac490c08a90d3788c57665115e3bc40ba\": container with ID starting with 55dd92bbd6385a591d52a22e1a34cf4ac490c08a90d3788c57665115e3bc40ba not found: ID does not exist" Jul 06 19:56:30.466184 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:30.466162 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-03cdb5ed-openshift-default-97bdfd8ff-2lprq"] Jul 06 19:56:31.875254 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:31.875216 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc5c4c99-d284-4441-984d-b03d39642786" path="/var/lib/kubelet/pods/bc5c4c99-d284-4441-984d-b03d39642786/volumes" Jul 06 19:56:42.643308 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.643265 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb"] Jul 06 19:56:42.643889 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.643874 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc5c4c99-d284-4441-984d-b03d39642786" containerName="istio-proxy" Jul 06 19:56:42.643942 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.643892 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc5c4c99-d284-4441-984d-b03d39642786" containerName="istio-proxy" Jul 06 19:56:42.643942 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.643906 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc5c4c99-d284-4441-984d-b03d39642786" containerName="istio-proxy" Jul 06 19:56:42.643942 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.643914 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc5c4c99-d284-4441-984d-b03d39642786" containerName="istio-proxy" Jul 06 19:56:42.644072 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.644061 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc5c4c99-d284-4441-984d-b03d39642786" containerName="istio-proxy" Jul 06 19:56:42.647404 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.647381 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.649384 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.649363 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-173a5950-openshift-default-dockercfg-k6b2t\"" Jul 06 19:56:42.662500 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.662470 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb"] Jul 06 19:56:42.769916 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.769878 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-workload-socket\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.770104 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.769923 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-credential-socket\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.770104 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.769949 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-envoy\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.770104 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.769982 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istiod-ca-cert\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.770104 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.770050 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj65t\" (UniqueName: \"kubernetes.io/projected/8a3feaf2-21e7-40c4-9c25-cc957d96870d-kube-api-access-mj65t\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.770104 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.770085 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-workload-certs\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.770277 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.770109 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-token\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.770277 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.770181 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-podinfo\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.770277 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.770230 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-data\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.871207 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.871176 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-workload-socket\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.871410 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.871214 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-credential-socket\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.871410 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.871240 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-envoy\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.871410 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.871266 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istiod-ca-cert\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.871410 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.871293 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj65t\" (UniqueName: \"kubernetes.io/projected/8a3feaf2-21e7-40c4-9c25-cc957d96870d-kube-api-access-mj65t\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.871410 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.871318 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-workload-certs\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.871410 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.871350 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-token\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.871763 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.871496 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-podinfo\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.871763 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.871540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-data\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.871763 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.871619 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-credential-socket\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.871895 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.871801 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-workload-certs\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.871997 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.871977 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-data\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.872086 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.872067 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-workload-socket\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.872128 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.872082 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istiod-ca-cert\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.873672 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.873643 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-envoy\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.874027 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.874007 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-podinfo\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.878720 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.878674 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-token\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.879468 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.879452 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj65t\" (UniqueName: \"kubernetes.io/projected/8a3feaf2-21e7-40c4-9c25-cc957d96870d-kube-api-access-mj65t\") pod \"e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:42.967766 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:42.967626 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:43.116993 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:43.116963 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb"] Jul 06 19:56:43.118965 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:56:43.118936 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3feaf2_21e7_40c4_9c25_cc957d96870d.slice/crio-a82f3bbf05c97b35650a2c966dd7eaf4ffa78f21df26afb884b2f0da8ff7dffd WatchSource:0}: Error finding container a82f3bbf05c97b35650a2c966dd7eaf4ffa78f21df26afb884b2f0da8ff7dffd: Status 404 returned error can't find the container with id a82f3bbf05c97b35650a2c966dd7eaf4ffa78f21df26afb884b2f0da8ff7dffd Jul 06 19:56:43.121171 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:43.121129 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:56:43.121242 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:43.121221 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:56:43.121280 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:43.121266 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:56:43.518933 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:43.518894 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" event={"ID":"8a3feaf2-21e7-40c4-9c25-cc957d96870d","Type":"ContainerStarted","Data":"680952c4dc31a6fb5f556e5dfc9e4fce8ea20a40d4e4fa61185f3eee51e2cd63"} Jul 06 19:56:43.519118 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:43.518963 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" event={"ID":"8a3feaf2-21e7-40c4-9c25-cc957d96870d","Type":"ContainerStarted","Data":"a82f3bbf05c97b35650a2c966dd7eaf4ffa78f21df26afb884b2f0da8ff7dffd"} Jul 06 19:56:43.539580 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:43.539522 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" podStartSLOduration=1.539505122 podStartE2EDuration="1.539505122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:56:43.53856255 +0000 UTC m=+2048.167516539" watchObservedRunningTime="2026-07-06 19:56:43.539505122 +0000 UTC m=+2048.168459088" Jul 06 19:56:43.968289 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:43.968248 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:44.974014 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:44.973982 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:45.529581 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:45.529552 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:45.530531 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:45.530511 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:58.599892 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:58.599861 2578 generic.go:356] "Generic (PLEG): container finished" podID="8a3feaf2-21e7-40c4-9c25-cc957d96870d" containerID="680952c4dc31a6fb5f556e5dfc9e4fce8ea20a40d4e4fa61185f3eee51e2cd63" exitCode=0 Jul 06 19:56:58.600291 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:58.599932 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" event={"ID":"8a3feaf2-21e7-40c4-9c25-cc957d96870d","Type":"ContainerDied","Data":"680952c4dc31a6fb5f556e5dfc9e4fce8ea20a40d4e4fa61185f3eee51e2cd63"} Jul 06 19:56:58.600385 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:58.600368 2578 scope.go:117] "RemoveContainer" containerID="680952c4dc31a6fb5f556e5dfc9e4fce8ea20a40d4e4fa61185f3eee51e2cd63" Jul 06 19:56:58.601356 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:58.601323 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:56:58.601452 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:58.601416 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:56:58.601516 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:58.601483 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:56:59.606562 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:59.606525 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" event={"ID":"8a3feaf2-21e7-40c4-9c25-cc957d96870d","Type":"ContainerStarted","Data":"51283389c216f9ff2163ca8606c4e375872eb791bbd4f4df3d6feed88f8ea0b0"} Jul 06 19:56:59.968862 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:59.968749 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:56:59.970052 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:59.970020 2578 patch_prober.go:28] interesting pod/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 06 19:56:59.970192 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:56:59.970087 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" podUID="8a3feaf2-21e7-40c4-9c25-cc957d96870d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 06 19:57:00.530286 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:57:00.530243 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:57:00.968306 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:57:00.968201 2578 patch_prober.go:28] interesting pod/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 06 19:57:00.968306 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:57:00.968257 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" podUID="8a3feaf2-21e7-40c4-9c25-cc957d96870d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 06 19:57:01.968257 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:57:01.968213 2578 patch_prober.go:28] interesting pod/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 06 19:57:01.968484 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:57:01.968278 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" podUID="8a3feaf2-21e7-40c4-9c25-cc957d96870d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 06 19:57:02.968920 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:57:02.968880 2578 patch_prober.go:28] interesting pod/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 06 19:57:02.969341 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:57:02.968958 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" podUID="8a3feaf2-21e7-40c4-9c25-cc957d96870d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 06 19:57:04.002039 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:57:04.002006 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:57:04.003248 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:57:04.003223 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:57:35.999814 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:57:35.999783 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 19:57:36.007050 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:57:36.007027 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 19:58:18.133217 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:18.133177 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb"] Jul 06 19:58:18.133889 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:18.133457 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" podUID="8a3feaf2-21e7-40c4-9c25-cc957d96870d" containerName="istio-proxy" containerID="cri-o://51283389c216f9ff2163ca8606c4e375872eb791bbd4f4df3d6feed88f8ea0b0" gracePeriod=30 Jul 06 19:58:19.005515 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:19.005474 2578 patch_prober.go:28] interesting pod/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 19:58:19.005714 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:19.005530 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" podUID="8a3feaf2-21e7-40c4-9c25-cc957d96870d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:58:23.406715 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.406621 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:58:23.480187 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.480157 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj65t\" (UniqueName: \"kubernetes.io/projected/8a3feaf2-21e7-40c4-9c25-cc957d96870d-kube-api-access-mj65t\") pod \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " Jul 06 19:58:23.480395 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.480199 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-podinfo\") pod \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " Jul 06 19:58:23.480395 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.480274 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-token\") pod \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " Jul 06 19:58:23.480395 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.480299 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-envoy\") pod \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " Jul 06 19:58:23.480585 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.480432 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-workload-certs\") pod \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " Jul 06 19:58:23.480585 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.480475 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-data\") pod \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " Jul 06 19:58:23.480585 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.480567 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-workload-socket\") pod \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " Jul 06 19:58:23.480775 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.480609 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istiod-ca-cert\") pod \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " Jul 06 19:58:23.481668 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.481397 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8a3feaf2-21e7-40c4-9c25-cc957d96870d" (UID: "8a3feaf2-21e7-40c4-9c25-cc957d96870d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:58:23.481668 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.481619 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8a3feaf2-21e7-40c4-9c25-cc957d96870d" (UID: "8a3feaf2-21e7-40c4-9c25-cc957d96870d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:58:23.482728 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.482556 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-data" (OuterVolumeSpecName: "istio-data") pod "8a3feaf2-21e7-40c4-9c25-cc957d96870d" (UID: "8a3feaf2-21e7-40c4-9c25-cc957d96870d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:58:23.482845 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.482816 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8a3feaf2-21e7-40c4-9c25-cc957d96870d" (UID: "8a3feaf2-21e7-40c4-9c25-cc957d96870d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:58:23.482910 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.482870 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-credential-socket\") pod \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\" (UID: \"8a3feaf2-21e7-40c4-9c25-cc957d96870d\") " Jul 06 19:58:23.482999 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.482969 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-token" (OuterVolumeSpecName: "istio-token") pod "8a3feaf2-21e7-40c4-9c25-cc957d96870d" (UID: "8a3feaf2-21e7-40c4-9c25-cc957d96870d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:58:23.483126 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.483092 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8a3feaf2-21e7-40c4-9c25-cc957d96870d" (UID: "8a3feaf2-21e7-40c4-9c25-cc957d96870d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:58:23.483298 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.483258 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8a3feaf2-21e7-40c4-9c25-cc957d96870d" (UID: "8a3feaf2-21e7-40c4-9c25-cc957d96870d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:58:23.483298 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.483268 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3feaf2-21e7-40c4-9c25-cc957d96870d-kube-api-access-mj65t" (OuterVolumeSpecName: "kube-api-access-mj65t") pod "8a3feaf2-21e7-40c4-9c25-cc957d96870d" (UID: "8a3feaf2-21e7-40c4-9c25-cc957d96870d"). InnerVolumeSpecName "kube-api-access-mj65t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:58:23.483465 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.483421 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-workload-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:58:23.483465 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.483447 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istiod-ca-cert\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:58:23.483465 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.483462 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-credential-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:58:23.483624 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.483476 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj65t\" (UniqueName: \"kubernetes.io/projected/8a3feaf2-21e7-40c4-9c25-cc957d96870d-kube-api-access-mj65t\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:58:23.483624 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.483569 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-podinfo\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:58:23.483624 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.483589 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-token\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:58:23.483624 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.483602 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-workload-certs\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:58:23.483624 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.483614 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-data\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:58:23.490755 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.490726 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8a3feaf2-21e7-40c4-9c25-cc957d96870d" (UID: "8a3feaf2-21e7-40c4-9c25-cc957d96870d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:58:23.584406 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:23.584356 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3feaf2-21e7-40c4-9c25-cc957d96870d-istio-envoy\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 19:58:24.053988 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:24.053952 2578 generic.go:356] "Generic (PLEG): container finished" podID="8a3feaf2-21e7-40c4-9c25-cc957d96870d" containerID="51283389c216f9ff2163ca8606c4e375872eb791bbd4f4df3d6feed88f8ea0b0" exitCode=0 Jul 06 19:58:24.054164 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:24.054016 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" Jul 06 19:58:24.054164 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:24.054030 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" event={"ID":"8a3feaf2-21e7-40c4-9c25-cc957d96870d","Type":"ContainerDied","Data":"51283389c216f9ff2163ca8606c4e375872eb791bbd4f4df3d6feed88f8ea0b0"} Jul 06 19:58:24.054164 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:24.054088 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb" event={"ID":"8a3feaf2-21e7-40c4-9c25-cc957d96870d","Type":"ContainerDied","Data":"a82f3bbf05c97b35650a2c966dd7eaf4ffa78f21df26afb884b2f0da8ff7dffd"} Jul 06 19:58:24.054164 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:24.054110 2578 scope.go:117] "RemoveContainer" containerID="51283389c216f9ff2163ca8606c4e375872eb791bbd4f4df3d6feed88f8ea0b0" Jul 06 19:58:24.063111 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:24.063091 2578 scope.go:117] "RemoveContainer" containerID="680952c4dc31a6fb5f556e5dfc9e4fce8ea20a40d4e4fa61185f3eee51e2cd63" Jul 06 19:58:24.073462 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:24.073441 2578 scope.go:117] "RemoveContainer" containerID="51283389c216f9ff2163ca8606c4e375872eb791bbd4f4df3d6feed88f8ea0b0" Jul 06 19:58:24.073803 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:58:24.073778 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51283389c216f9ff2163ca8606c4e375872eb791bbd4f4df3d6feed88f8ea0b0\": container with ID starting with 51283389c216f9ff2163ca8606c4e375872eb791bbd4f4df3d6feed88f8ea0b0 not found: ID does not exist" containerID="51283389c216f9ff2163ca8606c4e375872eb791bbd4f4df3d6feed88f8ea0b0" Jul 06 19:58:24.073929 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:24.073818 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51283389c216f9ff2163ca8606c4e375872eb791bbd4f4df3d6feed88f8ea0b0"} err="failed to get container status \"51283389c216f9ff2163ca8606c4e375872eb791bbd4f4df3d6feed88f8ea0b0\": rpc error: code = NotFound desc = could not find container \"51283389c216f9ff2163ca8606c4e375872eb791bbd4f4df3d6feed88f8ea0b0\": container with ID starting with 51283389c216f9ff2163ca8606c4e375872eb791bbd4f4df3d6feed88f8ea0b0 not found: ID does not exist" Jul 06 19:58:24.073929 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:24.073842 2578 scope.go:117] "RemoveContainer" containerID="680952c4dc31a6fb5f556e5dfc9e4fce8ea20a40d4e4fa61185f3eee51e2cd63" Jul 06 19:58:24.074164 ip-10-0-140-94 kubenswrapper[2578]: E0706 19:58:24.074137 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680952c4dc31a6fb5f556e5dfc9e4fce8ea20a40d4e4fa61185f3eee51e2cd63\": container with ID starting with 680952c4dc31a6fb5f556e5dfc9e4fce8ea20a40d4e4fa61185f3eee51e2cd63 not found: ID does not exist" containerID="680952c4dc31a6fb5f556e5dfc9e4fce8ea20a40d4e4fa61185f3eee51e2cd63" Jul 06 19:58:24.074213 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:24.074174 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680952c4dc31a6fb5f556e5dfc9e4fce8ea20a40d4e4fa61185f3eee51e2cd63"} err="failed to get container status \"680952c4dc31a6fb5f556e5dfc9e4fce8ea20a40d4e4fa61185f3eee51e2cd63\": rpc error: code = NotFound desc = could not find container \"680952c4dc31a6fb5f556e5dfc9e4fce8ea20a40d4e4fa61185f3eee51e2cd63\": container with ID starting with 680952c4dc31a6fb5f556e5dfc9e4fce8ea20a40d4e4fa61185f3eee51e2cd63 not found: ID does not exist" Jul 06 19:58:24.082726 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:24.082684 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb"] Jul 06 19:58:24.087965 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:24.087936 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-173a5950-openshift-default-5c74bf75f-rzzbb"] Jul 06 19:58:25.875539 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:58:25.875461 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a3feaf2-21e7-40c4-9c25-cc957d96870d" path="/var/lib/kubelet/pods/8a3feaf2-21e7-40c4-9c25-cc957d96870d/volumes" Jul 06 19:59:19.834328 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:19.834295 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4"] Jul 06 19:59:19.834821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:19.834684 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3feaf2-21e7-40c4-9c25-cc957d96870d" containerName="istio-proxy" Jul 06 19:59:19.834821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:19.834716 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3feaf2-21e7-40c4-9c25-cc957d96870d" containerName="istio-proxy" Jul 06 19:59:19.834821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:19.834805 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3feaf2-21e7-40c4-9c25-cc957d96870d" containerName="istio-proxy" Jul 06 19:59:19.834821 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:19.834813 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc5c4c99-d284-4441-984d-b03d39642786" containerName="istio-proxy" Jul 06 19:59:19.837948 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:19.837931 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:19.843473 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:19.843451 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-34eebf-openshift-default-dockercfg-glms4\"" Jul 06 19:59:19.863977 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:19.863930 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4"] Jul 06 19:59:19.989052 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:19.989015 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bebcad00-218f-46e7-9fec-2913059483c4-istio-token\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:19.989237 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:19.989074 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbb62\" (UniqueName: \"kubernetes.io/projected/bebcad00-218f-46e7-9fec-2913059483c4-kube-api-access-kbb62\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:19.989237 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:19.989135 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-istio-envoy\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:19.989237 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:19.989173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-credential-socket\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:19.989237 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:19.989197 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebcad00-218f-46e7-9fec-2913059483c4-istiod-ca-cert\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:19.989392 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:19.989252 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebcad00-218f-46e7-9fec-2913059483c4-istio-podinfo\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:19.989392 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:19.989315 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-istio-data\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:19.989392 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:19.989347 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-workload-certs\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:19.989392 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:19.989374 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-workload-socket\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.090644 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.090556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-istio-envoy\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.090644 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.090595 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-credential-socket\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.090878 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.090726 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebcad00-218f-46e7-9fec-2913059483c4-istiod-ca-cert\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.090878 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.090798 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebcad00-218f-46e7-9fec-2913059483c4-istio-podinfo\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.090878 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.090844 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-istio-data\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.091043 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.090879 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-workload-certs\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.091043 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.091003 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-credential-socket\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.091043 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.091007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-workload-socket\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.091205 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.091102 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bebcad00-218f-46e7-9fec-2913059483c4-istio-token\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.091205 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.091169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbb62\" (UniqueName: \"kubernetes.io/projected/bebcad00-218f-46e7-9fec-2913059483c4-kube-api-access-kbb62\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.091332 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.091238 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-workload-certs\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.091332 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.091274 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-istio-data\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.091332 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.091310 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-workload-socket\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.091565 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.091439 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebcad00-218f-46e7-9fec-2913059483c4-istiod-ca-cert\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.093076 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.093051 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-istio-envoy\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.093305 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.093286 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebcad00-218f-46e7-9fec-2913059483c4-istio-podinfo\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.103752 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.103730 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bebcad00-218f-46e7-9fec-2913059483c4-istio-token\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.105312 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.105287 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbb62\" (UniqueName: \"kubernetes.io/projected/bebcad00-218f-46e7-9fec-2913059483c4-kube-api-access-kbb62\") pod \"e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.149114 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.149079 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:20.309123 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.309090 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4"] Jul 06 19:59:20.312739 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:59:20.312713 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbebcad00_218f_46e7_9fec_2913059483c4.slice/crio-ca61737d2de36b1b5bf478d3f5ad38c10208762791c9485ec38835ffd7f68a35 WatchSource:0}: Error finding container ca61737d2de36b1b5bf478d3f5ad38c10208762791c9485ec38835ffd7f68a35: Status 404 returned error can't find the container with id ca61737d2de36b1b5bf478d3f5ad38c10208762791c9485ec38835ffd7f68a35 Jul 06 19:59:20.314853 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.314816 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:59:20.314953 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.314889 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:59:20.314953 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.314925 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:59:20.343778 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:20.343718 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" event={"ID":"bebcad00-218f-46e7-9fec-2913059483c4","Type":"ContainerStarted","Data":"ca61737d2de36b1b5bf478d3f5ad38c10208762791c9485ec38835ffd7f68a35"} Jul 06 19:59:21.350873 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:21.350838 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" event={"ID":"bebcad00-218f-46e7-9fec-2913059483c4","Type":"ContainerStarted","Data":"5441ad6026e95c0e1d3aff82405026637ef9358b0875915c41fa8a3dff89582a"} Jul 06 19:59:21.374873 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:21.374815 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" podStartSLOduration=2.3748004480000002 podStartE2EDuration="2.374800448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:59:21.373210619 +0000 UTC m=+2206.002164585" watchObservedRunningTime="2026-07-06 19:59:21.374800448 +0000 UTC m=+2206.003754415" Jul 06 19:59:22.149852 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:22.149812 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:22.154934 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:22.154907 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:22.356589 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:22.356558 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:22.357654 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:22.357634 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:28.541209 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.541172 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn"] Jul 06 19:59:28.541606 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.541593 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3feaf2-21e7-40c4-9c25-cc957d96870d" containerName="istio-proxy" Jul 06 19:59:28.541651 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.541608 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3feaf2-21e7-40c4-9c25-cc957d96870d" containerName="istio-proxy" Jul 06 19:59:28.541729 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.541719 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3feaf2-21e7-40c4-9c25-cc957d96870d" containerName="istio-proxy" Jul 06 19:59:28.546016 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.545996 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.548413 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.548392 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6bd627-openshift-default-dockercfg-55hbw\"" Jul 06 19:59:28.560170 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.560145 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn"] Jul 06 19:59:28.671994 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.671951 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af72417-3594-4c63-a4f1-faf99b704158-istio-podinfo\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.672174 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.672016 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af72417-3594-4c63-a4f1-faf99b704158-istio-token\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.672174 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.672085 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-workload-certs\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.672174 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.672151 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-workload-socket\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.672351 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.672181 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-istio-envoy\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.672351 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.672222 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-credential-socket\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.672351 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.672273 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af72417-3594-4c63-a4f1-faf99b704158-istiod-ca-cert\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.672351 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.672300 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-istio-data\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.672351 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.672342 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l74br\" (UniqueName: \"kubernetes.io/projected/9af72417-3594-4c63-a4f1-faf99b704158-kube-api-access-l74br\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.773788 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.773752 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-workload-socket\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.773788 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.773801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-istio-envoy\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.774057 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.773844 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-credential-socket\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.774057 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.773890 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af72417-3594-4c63-a4f1-faf99b704158-istiod-ca-cert\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.774057 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.773920 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-istio-data\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.774057 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.773961 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l74br\" (UniqueName: \"kubernetes.io/projected/9af72417-3594-4c63-a4f1-faf99b704158-kube-api-access-l74br\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.774057 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.774009 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af72417-3594-4c63-a4f1-faf99b704158-istio-podinfo\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.774057 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.774049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af72417-3594-4c63-a4f1-faf99b704158-istio-token\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.774374 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.774092 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-workload-certs\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.774374 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.774196 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-workload-socket\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.774374 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.774308 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-credential-socket\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.774552 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.774531 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-istio-data\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.774606 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.774567 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-workload-certs\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.774645 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.774601 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af72417-3594-4c63-a4f1-faf99b704158-istiod-ca-cert\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.776195 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.776162 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-istio-envoy\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.776685 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.776661 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af72417-3594-4c63-a4f1-faf99b704158-istio-podinfo\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.782779 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.782748 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l74br\" (UniqueName: \"kubernetes.io/projected/9af72417-3594-4c63-a4f1-faf99b704158-kube-api-access-l74br\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.783175 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.783153 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af72417-3594-4c63-a4f1-faf99b704158-istio-token\") pod \"e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:28.858077 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:28.857997 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:29.002978 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:29.002951 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn"] Jul 06 19:59:29.004634 ip-10-0-140-94 kubenswrapper[2578]: W0706 19:59:29.004602 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af72417_3594_4c63_a4f1_faf99b704158.slice/crio-119cba89b708b009f65fde448fe9f0836a546394beb66545d39311519e5b4153 WatchSource:0}: Error finding container 119cba89b708b009f65fde448fe9f0836a546394beb66545d39311519e5b4153: Status 404 returned error can't find the container with id 119cba89b708b009f65fde448fe9f0836a546394beb66545d39311519e5b4153 Jul 06 19:59:29.006809 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:29.006769 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:59:29.006888 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:29.006859 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:59:29.006940 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:29.006908 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:59:29.400675 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:29.400632 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" event={"ID":"9af72417-3594-4c63-a4f1-faf99b704158","Type":"ContainerStarted","Data":"037a68d30e618e9b5226d8a3401e1865c5ae318808e30d46bfe4db70737b9205"} Jul 06 19:59:29.400893 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:29.400737 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" event={"ID":"9af72417-3594-4c63-a4f1-faf99b704158","Type":"ContainerStarted","Data":"119cba89b708b009f65fde448fe9f0836a546394beb66545d39311519e5b4153"} Jul 06 19:59:29.424089 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:29.424025 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" podStartSLOduration=1.424009849 podStartE2EDuration="1.424009849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:59:29.422466234 +0000 UTC m=+2214.051420199" watchObservedRunningTime="2026-07-06 19:59:29.424009849 +0000 UTC m=+2214.052963815" Jul 06 19:59:29.858554 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:29.858505 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:30.864756 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:30.864718 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:31.411991 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:31.411957 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:31.412938 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:31.412917 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:37.671709 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:37.670346 2578 patch_prober.go:28] interesting pod/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": read tcp 10.132.0.2:35452->10.132.0.51:15021: read: connection reset by peer" start-of-body= Jul 06 19:59:37.671709 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:37.670421 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" podUID="bebcad00-218f-46e7-9fec-2913059483c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": read tcp 10.132.0.2:35452->10.132.0.51:15021: read: connection reset by peer" Jul 06 19:59:38.463155 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:38.463121 2578 generic.go:356] "Generic (PLEG): container finished" podID="bebcad00-218f-46e7-9fec-2913059483c4" containerID="5441ad6026e95c0e1d3aff82405026637ef9358b0875915c41fa8a3dff89582a" exitCode=0 Jul 06 19:59:38.463336 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:38.463198 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" event={"ID":"bebcad00-218f-46e7-9fec-2913059483c4","Type":"ContainerDied","Data":"5441ad6026e95c0e1d3aff82405026637ef9358b0875915c41fa8a3dff89582a"} Jul 06 19:59:38.463673 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:38.463658 2578 scope.go:117] "RemoveContainer" containerID="5441ad6026e95c0e1d3aff82405026637ef9358b0875915c41fa8a3dff89582a" Jul 06 19:59:38.464816 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:38.464784 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:59:38.464900 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:38.464862 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:59:38.464900 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:38.464895 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:59:39.474946 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:39.474910 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" event={"ID":"bebcad00-218f-46e7-9fec-2913059483c4","Type":"ContainerStarted","Data":"ebf95ed247db25de619b6c30301af547f411ccc696941c76a0084cb98e9335e2"} Jul 06 19:59:40.150555 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:40.150516 2578 patch_prober.go:28] interesting pod/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 06 19:59:40.150918 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:40.150875 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" podUID="bebcad00-218f-46e7-9fec-2913059483c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 06 19:59:40.151364 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:40.151348 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:41.149843 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:41.149798 2578 patch_prober.go:28] interesting pod/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 06 19:59:41.150482 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:41.150447 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" podUID="bebcad00-218f-46e7-9fec-2913059483c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 06 19:59:42.150810 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:42.150294 2578 patch_prober.go:28] interesting pod/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 06 19:59:42.150810 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:42.150382 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" podUID="bebcad00-218f-46e7-9fec-2913059483c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 06 19:59:43.150211 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:43.150156 2578 patch_prober.go:28] interesting pod/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 06 19:59:43.150496 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:43.150473 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" podUID="bebcad00-218f-46e7-9fec-2913059483c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 06 19:59:44.153650 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:44.153618 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:44.154134 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:44.153962 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:44.154741 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:44.154722 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 19:59:44.503036 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:44.502930 2578 generic.go:356] "Generic (PLEG): container finished" podID="9af72417-3594-4c63-a4f1-faf99b704158" containerID="037a68d30e618e9b5226d8a3401e1865c5ae318808e30d46bfe4db70737b9205" exitCode=0 Jul 06 19:59:44.503036 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:44.503008 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" event={"ID":"9af72417-3594-4c63-a4f1-faf99b704158","Type":"ContainerDied","Data":"037a68d30e618e9b5226d8a3401e1865c5ae318808e30d46bfe4db70737b9205"} Jul 06 19:59:44.503505 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:44.503484 2578 scope.go:117] "RemoveContainer" containerID="037a68d30e618e9b5226d8a3401e1865c5ae318808e30d46bfe4db70737b9205" Jul 06 19:59:44.504487 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:44.504458 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:59:44.504583 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:44.504523 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:59:44.504583 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:44.504558 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:59:45.510548 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:45.510512 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" event={"ID":"9af72417-3594-4c63-a4f1-faf99b704158","Type":"ContainerStarted","Data":"041ff739b15594f2fc21263af7de5f387805be18995933f3f52e5a314a7d670f"} Jul 06 19:59:45.859366 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:45.859329 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:45.860002 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:45.859973 2578 patch_prober.go:28] interesting pod/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 06 19:59:45.860146 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:45.860046 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" podUID="9af72417-3594-4c63-a4f1-faf99b704158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 06 19:59:46.412095 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:46.412051 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:46.858942 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:46.858903 2578 patch_prober.go:28] interesting pod/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 06 19:59:46.859368 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:46.858965 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" podUID="9af72417-3594-4c63-a4f1-faf99b704158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 06 19:59:47.858646 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:47.858605 2578 patch_prober.go:28] interesting pod/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 06 19:59:47.858887 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:47.858669 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" podUID="9af72417-3594-4c63-a4f1-faf99b704158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 06 19:59:48.859001 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:48.858959 2578 patch_prober.go:28] interesting pod/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 06 19:59:48.859510 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:48.859032 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" podUID="9af72417-3594-4c63-a4f1-faf99b704158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 06 19:59:49.863360 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:49.863328 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 19:59:49.864306 ip-10-0-140-94 kubenswrapper[2578]: I0706 19:59:49.864288 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 20:01:34.470255 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:34.470169 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4"] Jul 06 20:01:34.470736 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:34.470400 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" podUID="bebcad00-218f-46e7-9fec-2913059483c4" containerName="istio-proxy" containerID="cri-o://ebf95ed247db25de619b6c30301af547f411ccc696941c76a0084cb98e9335e2" gracePeriod=30 Jul 06 20:01:39.716568 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.716537 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 20:01:39.835862 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.835822 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbb62\" (UniqueName: \"kubernetes.io/projected/bebcad00-218f-46e7-9fec-2913059483c4-kube-api-access-kbb62\") pod \"bebcad00-218f-46e7-9fec-2913059483c4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " Jul 06 20:01:39.835862 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.835858 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bebcad00-218f-46e7-9fec-2913059483c4-istio-token\") pod \"bebcad00-218f-46e7-9fec-2913059483c4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " Jul 06 20:01:39.836133 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.835940 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-workload-certs\") pod \"bebcad00-218f-46e7-9fec-2913059483c4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " Jul 06 20:01:39.836133 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.835997 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-istio-envoy\") pod \"bebcad00-218f-46e7-9fec-2913059483c4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " Jul 06 20:01:39.836133 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.836023 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-credential-socket\") pod \"bebcad00-218f-46e7-9fec-2913059483c4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " Jul 06 20:01:39.836133 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.836048 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebcad00-218f-46e7-9fec-2913059483c4-istio-podinfo\") pod \"bebcad00-218f-46e7-9fec-2913059483c4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " Jul 06 20:01:39.836345 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.836191 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-istio-data\") pod \"bebcad00-218f-46e7-9fec-2913059483c4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " Jul 06 20:01:39.836345 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.836269 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-workload-socket\") pod \"bebcad00-218f-46e7-9fec-2913059483c4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " Jul 06 20:01:39.836345 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.836331 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bebcad00-218f-46e7-9fec-2913059483c4" (UID: "bebcad00-218f-46e7-9fec-2913059483c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:39.836510 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.836343 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bebcad00-218f-46e7-9fec-2913059483c4" (UID: "bebcad00-218f-46e7-9fec-2913059483c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:39.836510 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.836364 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebcad00-218f-46e7-9fec-2913059483c4-istiod-ca-cert\") pod \"bebcad00-218f-46e7-9fec-2913059483c4\" (UID: \"bebcad00-218f-46e7-9fec-2913059483c4\") " Jul 06 20:01:39.836613 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.836579 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bebcad00-218f-46e7-9fec-2913059483c4" (UID: "bebcad00-218f-46e7-9fec-2913059483c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:39.836748 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.836730 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-workload-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:01:39.836827 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.836751 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-workload-certs\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:01:39.836827 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.836766 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-credential-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:01:39.836827 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.836806 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bebcad00-218f-46e7-9fec-2913059483c4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bebcad00-218f-46e7-9fec-2913059483c4" (UID: "bebcad00-218f-46e7-9fec-2913059483c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:01:39.837234 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.837210 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-istio-data" (OuterVolumeSpecName: "istio-data") pod "bebcad00-218f-46e7-9fec-2913059483c4" (UID: "bebcad00-218f-46e7-9fec-2913059483c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:39.838137 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.838105 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bebcad00-218f-46e7-9fec-2913059483c4-kube-api-access-kbb62" (OuterVolumeSpecName: "kube-api-access-kbb62") pod "bebcad00-218f-46e7-9fec-2913059483c4" (UID: "bebcad00-218f-46e7-9fec-2913059483c4"). InnerVolumeSpecName "kube-api-access-kbb62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:01:39.838587 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.838565 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bebcad00-218f-46e7-9fec-2913059483c4" (UID: "bebcad00-218f-46e7-9fec-2913059483c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:39.838646 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.838584 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bebcad00-218f-46e7-9fec-2913059483c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "bebcad00-218f-46e7-9fec-2913059483c4" (UID: "bebcad00-218f-46e7-9fec-2913059483c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:01:39.838646 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.838583 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bebcad00-218f-46e7-9fec-2913059483c4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bebcad00-218f-46e7-9fec-2913059483c4" (UID: "bebcad00-218f-46e7-9fec-2913059483c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:01:39.937441 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.937409 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-istio-envoy\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:01:39.937441 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.937438 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebcad00-218f-46e7-9fec-2913059483c4-istio-podinfo\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:01:39.937441 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.937447 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebcad00-218f-46e7-9fec-2913059483c4-istio-data\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:01:39.937441 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.937454 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebcad00-218f-46e7-9fec-2913059483c4-istiod-ca-cert\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:01:39.937735 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.937463 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbb62\" (UniqueName: \"kubernetes.io/projected/bebcad00-218f-46e7-9fec-2913059483c4-kube-api-access-kbb62\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:01:39.937735 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:39.937472 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bebcad00-218f-46e7-9fec-2913059483c4-istio-token\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:01:40.123491 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:40.123404 2578 generic.go:356] "Generic (PLEG): container finished" podID="bebcad00-218f-46e7-9fec-2913059483c4" containerID="ebf95ed247db25de619b6c30301af547f411ccc696941c76a0084cb98e9335e2" exitCode=0 Jul 06 20:01:40.123491 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:40.123457 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" Jul 06 20:01:40.123791 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:40.123498 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" event={"ID":"bebcad00-218f-46e7-9fec-2913059483c4","Type":"ContainerDied","Data":"ebf95ed247db25de619b6c30301af547f411ccc696941c76a0084cb98e9335e2"} Jul 06 20:01:40.123791 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:40.123563 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4" event={"ID":"bebcad00-218f-46e7-9fec-2913059483c4","Type":"ContainerDied","Data":"ca61737d2de36b1b5bf478d3f5ad38c10208762791c9485ec38835ffd7f68a35"} Jul 06 20:01:40.123791 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:40.123593 2578 scope.go:117] "RemoveContainer" containerID="ebf95ed247db25de619b6c30301af547f411ccc696941c76a0084cb98e9335e2" Jul 06 20:01:40.133300 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:40.133284 2578 scope.go:117] "RemoveContainer" containerID="5441ad6026e95c0e1d3aff82405026637ef9358b0875915c41fa8a3dff89582a" Jul 06 20:01:40.141791 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:40.141771 2578 scope.go:117] "RemoveContainer" containerID="ebf95ed247db25de619b6c30301af547f411ccc696941c76a0084cb98e9335e2" Jul 06 20:01:40.142090 ip-10-0-140-94 kubenswrapper[2578]: E0706 20:01:40.142073 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf95ed247db25de619b6c30301af547f411ccc696941c76a0084cb98e9335e2\": container with ID starting with ebf95ed247db25de619b6c30301af547f411ccc696941c76a0084cb98e9335e2 not found: ID does not exist" containerID="ebf95ed247db25de619b6c30301af547f411ccc696941c76a0084cb98e9335e2" Jul 06 20:01:40.142147 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:40.142099 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf95ed247db25de619b6c30301af547f411ccc696941c76a0084cb98e9335e2"} err="failed to get container status \"ebf95ed247db25de619b6c30301af547f411ccc696941c76a0084cb98e9335e2\": rpc error: code = NotFound desc = could not find container \"ebf95ed247db25de619b6c30301af547f411ccc696941c76a0084cb98e9335e2\": container with ID starting with ebf95ed247db25de619b6c30301af547f411ccc696941c76a0084cb98e9335e2 not found: ID does not exist" Jul 06 20:01:40.142147 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:40.142116 2578 scope.go:117] "RemoveContainer" containerID="5441ad6026e95c0e1d3aff82405026637ef9358b0875915c41fa8a3dff89582a" Jul 06 20:01:40.142147 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:40.142133 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4"] Jul 06 20:01:40.142341 ip-10-0-140-94 kubenswrapper[2578]: E0706 20:01:40.142324 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5441ad6026e95c0e1d3aff82405026637ef9358b0875915c41fa8a3dff89582a\": container with ID starting with 5441ad6026e95c0e1d3aff82405026637ef9358b0875915c41fa8a3dff89582a not found: ID does not exist" containerID="5441ad6026e95c0e1d3aff82405026637ef9358b0875915c41fa8a3dff89582a" Jul 06 20:01:40.142382 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:40.142343 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5441ad6026e95c0e1d3aff82405026637ef9358b0875915c41fa8a3dff89582a"} err="failed to get container status \"5441ad6026e95c0e1d3aff82405026637ef9358b0875915c41fa8a3dff89582a\": rpc error: code = NotFound desc = could not find container \"5441ad6026e95c0e1d3aff82405026637ef9358b0875915c41fa8a3dff89582a\": container with ID starting with 5441ad6026e95c0e1d3aff82405026637ef9358b0875915c41fa8a3dff89582a not found: ID does not exist" Jul 06 20:01:40.148983 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:40.148962 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-34eebf-openshift-default-68b6c849c5-z4nn4"] Jul 06 20:01:41.876340 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:41.876307 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bebcad00-218f-46e7-9fec-2913059483c4" path="/var/lib/kubelet/pods/bebcad00-218f-46e7-9fec-2913059483c4/volumes" Jul 06 20:01:41.967226 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:41.967193 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn"] Jul 06 20:01:41.967458 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:41.967418 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" podUID="9af72417-3594-4c63-a4f1-faf99b704158" containerName="istio-proxy" containerID="cri-o://041ff739b15594f2fc21263af7de5f387805be18995933f3f52e5a314a7d670f" gracePeriod=30 Jul 06 20:01:42.856392 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.856357 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4"] Jul 06 20:01:42.856998 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.856983 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bebcad00-218f-46e7-9fec-2913059483c4" containerName="istio-proxy" Jul 06 20:01:42.857055 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.857004 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bebcad00-218f-46e7-9fec-2913059483c4" containerName="istio-proxy" Jul 06 20:01:42.857055 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.857031 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bebcad00-218f-46e7-9fec-2913059483c4" containerName="istio-proxy" Jul 06 20:01:42.857055 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.857039 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bebcad00-218f-46e7-9fec-2913059483c4" containerName="istio-proxy" Jul 06 20:01:42.857155 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.857134 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bebcad00-218f-46e7-9fec-2913059483c4" containerName="istio-proxy" Jul 06 20:01:42.857189 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.857155 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bebcad00-218f-46e7-9fec-2913059483c4" containerName="istio-proxy" Jul 06 20:01:42.862133 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.862112 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.862559 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.862536 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-token\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.862710 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.862580 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-credential-socket\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.862789 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.862684 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-data\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.862789 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.862749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-envoy\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.862900 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.862805 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-workload-certs\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.862900 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.862881 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-workload-socket\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.862997 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.862908 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-podinfo\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.862997 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.862956 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b8a342-1b14-4066-adce-012d8e1efaaf-istiod-ca-cert\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.862997 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.862983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v988w\" (UniqueName: \"kubernetes.io/projected/36b8a342-1b14-4066-adce-012d8e1efaaf-kube-api-access-v988w\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.864435 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.864385 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-aaefa8-openshift-default-dockercfg-x2728\"" Jul 06 20:01:42.878780 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.878756 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4"] Jul 06 20:01:42.963813 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.963774 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-workload-certs\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.964031 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.963856 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-workload-socket\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.964031 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.963885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-podinfo\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.964031 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.963936 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b8a342-1b14-4066-adce-012d8e1efaaf-istiod-ca-cert\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.964031 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.963964 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v988w\" (UniqueName: \"kubernetes.io/projected/36b8a342-1b14-4066-adce-012d8e1efaaf-kube-api-access-v988w\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.964031 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.963997 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-token\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.964306 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.964049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-credential-socket\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.964306 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.964105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-data\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.964306 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.964146 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-envoy\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.964306 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.964267 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-workload-certs\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.965077 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.964958 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-workload-socket\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.965631 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.965596 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b8a342-1b14-4066-adce-012d8e1efaaf-istiod-ca-cert\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.966201 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.965585 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-data\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.966332 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.966303 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-credential-socket\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.967247 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.967226 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-envoy\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.967998 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.967980 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-podinfo\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.973048 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.973026 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-token\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:42.973450 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:42.973429 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v988w\" (UniqueName: \"kubernetes.io/projected/36b8a342-1b14-4066-adce-012d8e1efaaf-kube-api-access-v988w\") pod \"e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:43.173099 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:43.172998 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:43.339409 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:43.339268 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4"] Jul 06 20:01:43.342373 ip-10-0-140-94 kubenswrapper[2578]: W0706 20:01:43.342344 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b8a342_1b14_4066_adce_012d8e1efaaf.slice/crio-8ac98b3b3cdcf9acc26a623e2b33e6e77b786cce73ee0dba1656e8b5a8d8ad03 WatchSource:0}: Error finding container 8ac98b3b3cdcf9acc26a623e2b33e6e77b786cce73ee0dba1656e8b5a8d8ad03: Status 404 returned error can't find the container with id 8ac98b3b3cdcf9acc26a623e2b33e6e77b786cce73ee0dba1656e8b5a8d8ad03 Jul 06 20:01:43.345039 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:43.344999 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:01:43.345126 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:43.345082 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:01:43.345169 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:43.345131 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:01:44.148242 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:44.148202 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" event={"ID":"36b8a342-1b14-4066-adce-012d8e1efaaf","Type":"ContainerStarted","Data":"a53086157af5a4e1003aa6936ffb57a2f32129571cba21c9046e42fb946ae6d1"} Jul 06 20:01:44.148756 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:44.148260 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" event={"ID":"36b8a342-1b14-4066-adce-012d8e1efaaf","Type":"ContainerStarted","Data":"8ac98b3b3cdcf9acc26a623e2b33e6e77b786cce73ee0dba1656e8b5a8d8ad03"} Jul 06 20:01:44.170651 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:44.170599 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" podStartSLOduration=2.170581393 podStartE2EDuration="2.170581393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:01:44.168274538 +0000 UTC m=+2348.797228505" watchObservedRunningTime="2026-07-06 20:01:44.170581393 +0000 UTC m=+2348.799535433" Jul 06 20:01:44.173625 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:44.173601 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:44.179194 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:44.179172 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:45.154141 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:45.154109 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:45.155265 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:45.155244 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:01:47.167609 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.167573 2578 generic.go:356] "Generic (PLEG): container finished" podID="9af72417-3594-4c63-a4f1-faf99b704158" containerID="041ff739b15594f2fc21263af7de5f387805be18995933f3f52e5a314a7d670f" exitCode=0 Jul 06 20:01:47.168093 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.167645 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" event={"ID":"9af72417-3594-4c63-a4f1-faf99b704158","Type":"ContainerDied","Data":"041ff739b15594f2fc21263af7de5f387805be18995933f3f52e5a314a7d670f"} Jul 06 20:01:47.168093 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.167757 2578 scope.go:117] "RemoveContainer" containerID="037a68d30e618e9b5226d8a3401e1865c5ae318808e30d46bfe4db70737b9205" Jul 06 20:01:47.238078 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.237941 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 20:01:47.307036 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.306998 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af72417-3594-4c63-a4f1-faf99b704158-istio-podinfo\") pod \"9af72417-3594-4c63-a4f1-faf99b704158\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " Jul 06 20:01:47.307198 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.307050 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af72417-3594-4c63-a4f1-faf99b704158-istio-token\") pod \"9af72417-3594-4c63-a4f1-faf99b704158\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " Jul 06 20:01:47.307198 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.307084 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af72417-3594-4c63-a4f1-faf99b704158-istiod-ca-cert\") pod \"9af72417-3594-4c63-a4f1-faf99b704158\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " Jul 06 20:01:47.307198 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.307155 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-istio-envoy\") pod \"9af72417-3594-4c63-a4f1-faf99b704158\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " Jul 06 20:01:47.307198 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.307190 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l74br\" (UniqueName: \"kubernetes.io/projected/9af72417-3594-4c63-a4f1-faf99b704158-kube-api-access-l74br\") pod \"9af72417-3594-4c63-a4f1-faf99b704158\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " Jul 06 20:01:47.307500 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.307340 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-istio-data\") pod \"9af72417-3594-4c63-a4f1-faf99b704158\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " Jul 06 20:01:47.307500 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.307382 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-workload-certs\") pod \"9af72417-3594-4c63-a4f1-faf99b704158\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " Jul 06 20:01:47.307500 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.307431 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-workload-socket\") pod \"9af72417-3594-4c63-a4f1-faf99b704158\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " Jul 06 20:01:47.307500 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.307472 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-credential-socket\") pod \"9af72417-3594-4c63-a4f1-faf99b704158\" (UID: \"9af72417-3594-4c63-a4f1-faf99b704158\") " Jul 06 20:01:47.308264 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.308080 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af72417-3594-4c63-a4f1-faf99b704158-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9af72417-3594-4c63-a4f1-faf99b704158" (UID: "9af72417-3594-4c63-a4f1-faf99b704158"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:01:47.308386 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.308278 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9af72417-3594-4c63-a4f1-faf99b704158" (UID: "9af72417-3594-4c63-a4f1-faf99b704158"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:47.308386 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.308307 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9af72417-3594-4c63-a4f1-faf99b704158" (UID: "9af72417-3594-4c63-a4f1-faf99b704158"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:47.308504 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.308460 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9af72417-3594-4c63-a4f1-faf99b704158" (UID: "9af72417-3594-4c63-a4f1-faf99b704158"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:47.308938 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.308894 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-istio-data" (OuterVolumeSpecName: "istio-data") pod "9af72417-3594-4c63-a4f1-faf99b704158" (UID: "9af72417-3594-4c63-a4f1-faf99b704158"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:47.310523 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.310495 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af72417-3594-4c63-a4f1-faf99b704158-kube-api-access-l74br" (OuterVolumeSpecName: "kube-api-access-l74br") pod "9af72417-3594-4c63-a4f1-faf99b704158" (UID: "9af72417-3594-4c63-a4f1-faf99b704158"). InnerVolumeSpecName "kube-api-access-l74br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:01:47.310632 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.310582 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af72417-3594-4c63-a4f1-faf99b704158-istio-token" (OuterVolumeSpecName: "istio-token") pod "9af72417-3594-4c63-a4f1-faf99b704158" (UID: "9af72417-3594-4c63-a4f1-faf99b704158"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:01:47.310802 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.310709 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9af72417-3594-4c63-a4f1-faf99b704158" (UID: "9af72417-3594-4c63-a4f1-faf99b704158"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:47.311141 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.311111 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9af72417-3594-4c63-a4f1-faf99b704158-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9af72417-3594-4c63-a4f1-faf99b704158" (UID: "9af72417-3594-4c63-a4f1-faf99b704158"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:01:47.408804 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.408765 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-workload-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:01:47.408804 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.408796 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-credential-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:01:47.408804 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.408806 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af72417-3594-4c63-a4f1-faf99b704158-istio-podinfo\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:01:47.408804 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.408814 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af72417-3594-4c63-a4f1-faf99b704158-istio-token\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:01:47.409082 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.408822 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af72417-3594-4c63-a4f1-faf99b704158-istiod-ca-cert\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:01:47.409082 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.408830 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-istio-envoy\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:01:47.409082 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.408841 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l74br\" (UniqueName: \"kubernetes.io/projected/9af72417-3594-4c63-a4f1-faf99b704158-kube-api-access-l74br\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:01:47.409082 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.408849 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-istio-data\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:01:47.409082 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:47.408857 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af72417-3594-4c63-a4f1-faf99b704158-workload-certs\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:01:48.173887 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:48.173810 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" Jul 06 20:01:48.174279 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:48.173810 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn" event={"ID":"9af72417-3594-4c63-a4f1-faf99b704158","Type":"ContainerDied","Data":"119cba89b708b009f65fde448fe9f0836a546394beb66545d39311519e5b4153"} Jul 06 20:01:48.174279 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:48.173937 2578 scope.go:117] "RemoveContainer" containerID="041ff739b15594f2fc21263af7de5f387805be18995933f3f52e5a314a7d670f" Jul 06 20:01:48.220990 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:48.220953 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn"] Jul 06 20:01:48.234066 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:48.234029 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6bd627-openshift-default-5b69967865-64vjn"] Jul 06 20:01:49.875168 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:49.875140 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9af72417-3594-4c63-a4f1-faf99b704158" path="/var/lib/kubelet/pods/9af72417-3594-4c63-a4f1-faf99b704158/volumes" Jul 06 20:01:51.408581 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.408547 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch"] Jul 06 20:01:51.409205 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.409186 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9af72417-3594-4c63-a4f1-faf99b704158" containerName="istio-proxy" Jul 06 20:01:51.409308 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.409208 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9af72417-3594-4c63-a4f1-faf99b704158" containerName="istio-proxy" Jul 06 20:01:51.409308 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.409225 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9af72417-3594-4c63-a4f1-faf99b704158" containerName="istio-proxy" Jul 06 20:01:51.409308 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.409232 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9af72417-3594-4c63-a4f1-faf99b704158" containerName="istio-proxy" Jul 06 20:01:51.409458 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.409349 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9af72417-3594-4c63-a4f1-faf99b704158" containerName="istio-proxy" Jul 06 20:01:51.409458 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.409365 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9af72417-3594-4c63-a4f1-faf99b704158" containerName="istio-proxy" Jul 06 20:01:51.413852 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.413830 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.416128 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.416107 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a82399-openshift-default-dockercfg-j97f5\"" Jul 06 20:01:51.431289 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.431265 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch"] Jul 06 20:01:51.442243 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.442219 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-podinfo\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.442385 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.442250 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istiod-ca-cert\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.442385 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.442274 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-envoy\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.442385 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.442290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-token\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.442385 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.442371 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-workload-certs\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.442615 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.442451 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-credential-socket\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.442615 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.442491 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-287px\" (UniqueName: \"kubernetes.io/projected/c1074b6d-ee33-49a9-a885-b836cbbfc19f-kube-api-access-287px\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.442615 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.442519 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-workload-socket\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.442769 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.442630 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-data\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.543303 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.543264 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-data\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.543303 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.543306 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-podinfo\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.543584 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.543328 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istiod-ca-cert\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.543584 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.543349 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-envoy\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.543584 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.543366 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-token\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.543584 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.543408 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-workload-certs\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.543584 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.543468 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-credential-socket\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.543584 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.543495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-287px\" (UniqueName: \"kubernetes.io/projected/c1074b6d-ee33-49a9-a885-b836cbbfc19f-kube-api-access-287px\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.543584 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.543519 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-workload-socket\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.543963 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.543830 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-data\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.543963 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.543942 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-credential-socket\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.544047 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.544000 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-workload-socket\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.544119 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.544101 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-workload-certs\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.544512 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.544491 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istiod-ca-cert\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.545803 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.545779 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-envoy\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.546300 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.546279 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-podinfo\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.557791 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.557770 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-token\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.558015 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.557991 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-287px\" (UniqueName: \"kubernetes.io/projected/c1074b6d-ee33-49a9-a885-b836cbbfc19f-kube-api-access-287px\") pod \"e2e-api-b-a82399-openshift-default-74c459cf65-292ch\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.725019 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.724929 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:51.945773 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.945749 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch"] Jul 06 20:01:51.947479 ip-10-0-140-94 kubenswrapper[2578]: W0706 20:01:51.947453 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1074b6d_ee33_49a9_a885_b836cbbfc19f.slice/crio-27bcee163e01a78f5cf6d8f9b46b60d9ddadd1d08a0c503a287e542aab7c5c6c WatchSource:0}: Error finding container 27bcee163e01a78f5cf6d8f9b46b60d9ddadd1d08a0c503a287e542aab7c5c6c: Status 404 returned error can't find the container with id 27bcee163e01a78f5cf6d8f9b46b60d9ddadd1d08a0c503a287e542aab7c5c6c Jul 06 20:01:51.949584 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.949550 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:01:51.949708 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.949619 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:01:51.949708 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:51.949647 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:01:52.200285 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:52.200253 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" event={"ID":"c1074b6d-ee33-49a9-a885-b836cbbfc19f","Type":"ContainerStarted","Data":"29d0471223db1e472d46c6c7b6058ed9c1bce9b1a309041a075041f024ee451e"} Jul 06 20:01:52.200457 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:52.200305 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" event={"ID":"c1074b6d-ee33-49a9-a885-b836cbbfc19f","Type":"ContainerStarted","Data":"27bcee163e01a78f5cf6d8f9b46b60d9ddadd1d08a0c503a287e542aab7c5c6c"} Jul 06 20:01:52.224812 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:52.224757 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" podStartSLOduration=1.224739745 podStartE2EDuration="1.224739745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:01:52.224322744 +0000 UTC m=+2356.853276712" watchObservedRunningTime="2026-07-06 20:01:52.224739745 +0000 UTC m=+2356.853693711" Jul 06 20:01:52.726004 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:52.725968 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:53.731209 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:53.731182 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:54.211623 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:54.211596 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:01:54.212681 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:01:54.212658 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:02:01.196879 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:01.196832 2578 patch_prober.go:28] interesting pod/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 20:02:01.197293 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:01.196912 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" podUID="36b8a342-1b14-4066-adce-012d8e1efaaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 20:02:02.263545 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:02.263442 2578 generic.go:356] "Generic (PLEG): container finished" podID="36b8a342-1b14-4066-adce-012d8e1efaaf" containerID="a53086157af5a4e1003aa6936ffb57a2f32129571cba21c9046e42fb946ae6d1" exitCode=0 Jul 06 20:02:02.263545 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:02.263519 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" event={"ID":"36b8a342-1b14-4066-adce-012d8e1efaaf","Type":"ContainerDied","Data":"a53086157af5a4e1003aa6936ffb57a2f32129571cba21c9046e42fb946ae6d1"} Jul 06 20:02:02.264172 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:02.264072 2578 scope.go:117] "RemoveContainer" containerID="a53086157af5a4e1003aa6936ffb57a2f32129571cba21c9046e42fb946ae6d1" Jul 06 20:02:02.265212 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:02.265180 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:02:02.265285 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:02.265255 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:02:02.265327 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:02.265289 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:02:03.272294 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:03.272255 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" event={"ID":"36b8a342-1b14-4066-adce-012d8e1efaaf","Type":"ContainerStarted","Data":"b5f89fbd0447be4966bdb6c04f04ac62ce5372a571178f1d6621e6a29fce4749"} Jul 06 20:02:04.174242 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:04.174195 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:02:04.174654 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:04.174624 2578 patch_prober.go:28] interesting pod/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 06 20:02:04.174812 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:04.174685 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" podUID="36b8a342-1b14-4066-adce-012d8e1efaaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 06 20:02:05.174482 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:05.174431 2578 patch_prober.go:28] interesting pod/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 06 20:02:05.175093 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:05.174544 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" podUID="36b8a342-1b14-4066-adce-012d8e1efaaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 06 20:02:06.174479 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:06.174428 2578 patch_prober.go:28] interesting pod/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 06 20:02:06.174731 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:06.174498 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" podUID="36b8a342-1b14-4066-adce-012d8e1efaaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 06 20:02:07.174364 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:07.174269 2578 patch_prober.go:28] interesting pod/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 06 20:02:07.174522 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:07.174369 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" podUID="36b8a342-1b14-4066-adce-012d8e1efaaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 06 20:02:08.177439 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:08.177405 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:02:08.177896 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:08.177741 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:02:08.178442 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:08.178426 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:02:09.850608 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:09.850556 2578 patch_prober.go:28] interesting pod/e2e-api-b-a82399-openshift-default-74c459cf65-292ch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:47226->10.132.0.54:15021: read: connection reset by peer" start-of-body= Jul 06 20:02:09.851090 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:09.850649 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" podUID="c1074b6d-ee33-49a9-a885-b836cbbfc19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:47226->10.132.0.54:15021: read: connection reset by peer" Jul 06 20:02:10.318826 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:10.318793 2578 generic.go:356] "Generic (PLEG): container finished" podID="c1074b6d-ee33-49a9-a885-b836cbbfc19f" containerID="29d0471223db1e472d46c6c7b6058ed9c1bce9b1a309041a075041f024ee451e" exitCode=0 Jul 06 20:02:10.319055 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:10.318855 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" event={"ID":"c1074b6d-ee33-49a9-a885-b836cbbfc19f","Type":"ContainerDied","Data":"29d0471223db1e472d46c6c7b6058ed9c1bce9b1a309041a075041f024ee451e"} Jul 06 20:02:10.319259 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:10.319243 2578 scope.go:117] "RemoveContainer" containerID="29d0471223db1e472d46c6c7b6058ed9c1bce9b1a309041a075041f024ee451e" Jul 06 20:02:10.320257 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:10.320228 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:02:10.320322 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:10.320296 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:02:10.320365 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:10.320328 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:02:11.325959 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:11.325912 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" event={"ID":"c1074b6d-ee33-49a9-a885-b836cbbfc19f","Type":"ContainerStarted","Data":"5c19e4c7dc146c4b8b74c18532aa681c0db8b6049c7cba516db39bf98e31023f"} Jul 06 20:02:11.726122 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:11.726020 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:02:11.726386 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:11.726352 2578 patch_prober.go:28] interesting pod/e2e-api-b-a82399-openshift-default-74c459cf65-292ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 06 20:02:11.726492 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:11.726409 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" podUID="c1074b6d-ee33-49a9-a885-b836cbbfc19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 06 20:02:12.725869 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:12.725832 2578 patch_prober.go:28] interesting pod/e2e-api-b-a82399-openshift-default-74c459cf65-292ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 06 20:02:12.726388 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:12.725895 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" podUID="c1074b6d-ee33-49a9-a885-b836cbbfc19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 06 20:02:13.726397 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:13.726358 2578 patch_prober.go:28] interesting pod/e2e-api-b-a82399-openshift-default-74c459cf65-292ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 06 20:02:13.726841 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:13.726421 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" podUID="c1074b6d-ee33-49a9-a885-b836cbbfc19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 06 20:02:13.808034 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:13.808001 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4"] Jul 06 20:02:13.808236 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:13.808215 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" podUID="36b8a342-1b14-4066-adce-012d8e1efaaf" containerName="istio-proxy" containerID="cri-o://b5f89fbd0447be4966bdb6c04f04ac62ce5372a571178f1d6621e6a29fce4749" gracePeriod=30 Jul 06 20:02:14.726291 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:14.726251 2578 patch_prober.go:28] interesting pod/e2e-api-b-a82399-openshift-default-74c459cf65-292ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 06 20:02:14.726477 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:14.726310 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" podUID="c1074b6d-ee33-49a9-a885-b836cbbfc19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 06 20:02:15.764535 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:15.764496 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:02:15.765132 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:15.765087 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:02:15.766312 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:15.766287 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:02:19.069343 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.069277 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:02:19.226571 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.226540 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v988w\" (UniqueName: \"kubernetes.io/projected/36b8a342-1b14-4066-adce-012d8e1efaaf-kube-api-access-v988w\") pod \"36b8a342-1b14-4066-adce-012d8e1efaaf\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " Jul 06 20:02:19.226780 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.226581 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-workload-certs\") pod \"36b8a342-1b14-4066-adce-012d8e1efaaf\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " Jul 06 20:02:19.226780 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.226604 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-workload-socket\") pod \"36b8a342-1b14-4066-adce-012d8e1efaaf\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " Jul 06 20:02:19.226780 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.226628 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-token\") pod \"36b8a342-1b14-4066-adce-012d8e1efaaf\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " Jul 06 20:02:19.226780 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.226656 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-envoy\") pod \"36b8a342-1b14-4066-adce-012d8e1efaaf\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " Jul 06 20:02:19.226780 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.226687 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-credential-socket\") pod \"36b8a342-1b14-4066-adce-012d8e1efaaf\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " Jul 06 20:02:19.226780 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.226760 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-data\") pod \"36b8a342-1b14-4066-adce-012d8e1efaaf\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " Jul 06 20:02:19.227082 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.226837 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-podinfo\") pod \"36b8a342-1b14-4066-adce-012d8e1efaaf\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " Jul 06 20:02:19.227082 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.226868 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b8a342-1b14-4066-adce-012d8e1efaaf-istiod-ca-cert\") pod \"36b8a342-1b14-4066-adce-012d8e1efaaf\" (UID: \"36b8a342-1b14-4066-adce-012d8e1efaaf\") " Jul 06 20:02:19.227082 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.226900 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "36b8a342-1b14-4066-adce-012d8e1efaaf" (UID: "36b8a342-1b14-4066-adce-012d8e1efaaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:19.227082 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.226982 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "36b8a342-1b14-4066-adce-012d8e1efaaf" (UID: "36b8a342-1b14-4066-adce-012d8e1efaaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:19.227297 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.227161 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "36b8a342-1b14-4066-adce-012d8e1efaaf" (UID: "36b8a342-1b14-4066-adce-012d8e1efaaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:19.227297 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.227252 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-workload-certs\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:02:19.227297 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.227272 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-workload-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:02:19.227297 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.227285 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-credential-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:02:19.227498 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.227353 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b8a342-1b14-4066-adce-012d8e1efaaf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "36b8a342-1b14-4066-adce-012d8e1efaaf" (UID: "36b8a342-1b14-4066-adce-012d8e1efaaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:02:19.227800 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.227770 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-data" (OuterVolumeSpecName: "istio-data") pod "36b8a342-1b14-4066-adce-012d8e1efaaf" (UID: "36b8a342-1b14-4066-adce-012d8e1efaaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:19.229097 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.229062 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-token" (OuterVolumeSpecName: "istio-token") pod "36b8a342-1b14-4066-adce-012d8e1efaaf" (UID: "36b8a342-1b14-4066-adce-012d8e1efaaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:02:19.229229 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.229091 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "36b8a342-1b14-4066-adce-012d8e1efaaf" (UID: "36b8a342-1b14-4066-adce-012d8e1efaaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:02:19.229333 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.229309 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "36b8a342-1b14-4066-adce-012d8e1efaaf" (UID: "36b8a342-1b14-4066-adce-012d8e1efaaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:19.229392 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.229377 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b8a342-1b14-4066-adce-012d8e1efaaf-kube-api-access-v988w" (OuterVolumeSpecName: "kube-api-access-v988w") pod "36b8a342-1b14-4066-adce-012d8e1efaaf" (UID: "36b8a342-1b14-4066-adce-012d8e1efaaf"). InnerVolumeSpecName "kube-api-access-v988w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:02:19.328278 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.328188 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v988w\" (UniqueName: \"kubernetes.io/projected/36b8a342-1b14-4066-adce-012d8e1efaaf-kube-api-access-v988w\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:02:19.328278 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.328220 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-token\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:02:19.328278 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.328231 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-envoy\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:02:19.328278 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.328241 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-data\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:02:19.328278 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.328249 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b8a342-1b14-4066-adce-012d8e1efaaf-istio-podinfo\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:02:19.328278 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.328256 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b8a342-1b14-4066-adce-012d8e1efaaf-istiod-ca-cert\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:02:19.383552 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.383512 2578 generic.go:356] "Generic (PLEG): container finished" podID="36b8a342-1b14-4066-adce-012d8e1efaaf" containerID="b5f89fbd0447be4966bdb6c04f04ac62ce5372a571178f1d6621e6a29fce4749" exitCode=0 Jul 06 20:02:19.383762 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.383572 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" Jul 06 20:02:19.383762 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.383604 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" event={"ID":"36b8a342-1b14-4066-adce-012d8e1efaaf","Type":"ContainerDied","Data":"b5f89fbd0447be4966bdb6c04f04ac62ce5372a571178f1d6621e6a29fce4749"} Jul 06 20:02:19.383762 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.383680 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4" event={"ID":"36b8a342-1b14-4066-adce-012d8e1efaaf","Type":"ContainerDied","Data":"8ac98b3b3cdcf9acc26a623e2b33e6e77b786cce73ee0dba1656e8b5a8d8ad03"} Jul 06 20:02:19.383762 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.383732 2578 scope.go:117] "RemoveContainer" containerID="b5f89fbd0447be4966bdb6c04f04ac62ce5372a571178f1d6621e6a29fce4749" Jul 06 20:02:19.397604 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.397582 2578 scope.go:117] "RemoveContainer" containerID="a53086157af5a4e1003aa6936ffb57a2f32129571cba21c9046e42fb946ae6d1" Jul 06 20:02:19.408270 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.408248 2578 scope.go:117] "RemoveContainer" containerID="b5f89fbd0447be4966bdb6c04f04ac62ce5372a571178f1d6621e6a29fce4749" Jul 06 20:02:19.408595 ip-10-0-140-94 kubenswrapper[2578]: E0706 20:02:19.408566 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5f89fbd0447be4966bdb6c04f04ac62ce5372a571178f1d6621e6a29fce4749\": container with ID starting with b5f89fbd0447be4966bdb6c04f04ac62ce5372a571178f1d6621e6a29fce4749 not found: ID does not exist" containerID="b5f89fbd0447be4966bdb6c04f04ac62ce5372a571178f1d6621e6a29fce4749" Jul 06 20:02:19.408657 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.408610 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5f89fbd0447be4966bdb6c04f04ac62ce5372a571178f1d6621e6a29fce4749"} err="failed to get container status \"b5f89fbd0447be4966bdb6c04f04ac62ce5372a571178f1d6621e6a29fce4749\": rpc error: code = NotFound desc = could not find container \"b5f89fbd0447be4966bdb6c04f04ac62ce5372a571178f1d6621e6a29fce4749\": container with ID starting with b5f89fbd0447be4966bdb6c04f04ac62ce5372a571178f1d6621e6a29fce4749 not found: ID does not exist" Jul 06 20:02:19.408657 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.408635 2578 scope.go:117] "RemoveContainer" containerID="a53086157af5a4e1003aa6936ffb57a2f32129571cba21c9046e42fb946ae6d1" Jul 06 20:02:19.408966 ip-10-0-140-94 kubenswrapper[2578]: E0706 20:02:19.408943 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a53086157af5a4e1003aa6936ffb57a2f32129571cba21c9046e42fb946ae6d1\": container with ID starting with a53086157af5a4e1003aa6936ffb57a2f32129571cba21c9046e42fb946ae6d1 not found: ID does not exist" containerID="a53086157af5a4e1003aa6936ffb57a2f32129571cba21c9046e42fb946ae6d1" Jul 06 20:02:19.409081 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.408968 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53086157af5a4e1003aa6936ffb57a2f32129571cba21c9046e42fb946ae6d1"} err="failed to get container status \"a53086157af5a4e1003aa6936ffb57a2f32129571cba21c9046e42fb946ae6d1\": rpc error: code = NotFound desc = could not find container \"a53086157af5a4e1003aa6936ffb57a2f32129571cba21c9046e42fb946ae6d1\": container with ID starting with a53086157af5a4e1003aa6936ffb57a2f32129571cba21c9046e42fb946ae6d1 not found: ID does not exist" Jul 06 20:02:19.415269 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.415244 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4"] Jul 06 20:02:19.419563 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.419541 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-aaefa8-openshift-default-67f86f89b7-vsbw4"] Jul 06 20:02:19.876074 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:19.876044 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36b8a342-1b14-4066-adce-012d8e1efaaf" path="/var/lib/kubelet/pods/36b8a342-1b14-4066-adce-012d8e1efaaf/volumes" Jul 06 20:02:21.215685 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:21.215649 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch"] Jul 06 20:02:21.216185 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:21.215903 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" podUID="c1074b6d-ee33-49a9-a885-b836cbbfc19f" containerName="istio-proxy" containerID="cri-o://5c19e4c7dc146c4b8b74c18532aa681c0db8b6049c7cba516db39bf98e31023f" gracePeriod=30 Jul 06 20:02:22.113647 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.113616 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx"] Jul 06 20:02:22.114040 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.114026 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36b8a342-1b14-4066-adce-012d8e1efaaf" containerName="istio-proxy" Jul 06 20:02:22.114099 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.114043 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="36b8a342-1b14-4066-adce-012d8e1efaaf" containerName="istio-proxy" Jul 06 20:02:22.114099 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.114055 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36b8a342-1b14-4066-adce-012d8e1efaaf" containerName="istio-proxy" Jul 06 20:02:22.114099 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.114061 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="36b8a342-1b14-4066-adce-012d8e1efaaf" containerName="istio-proxy" Jul 06 20:02:22.114211 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.114145 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="36b8a342-1b14-4066-adce-012d8e1efaaf" containerName="istio-proxy" Jul 06 20:02:22.114211 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.114156 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="36b8a342-1b14-4066-adce-012d8e1efaaf" containerName="istio-proxy" Jul 06 20:02:22.119039 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.119020 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.122366 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.122345 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-c21a78-openshift-default-dockercfg-w7gch\"" Jul 06 20:02:22.138551 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.138532 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx"] Jul 06 20:02:22.256190 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.256152 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-workload-certs\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.256664 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.256200 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946a47c4-df05-4578-a544-05eb1e8672cd-istio-podinfo\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.256664 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.256343 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-workload-socket\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.256664 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.256394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-istio-envoy\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.256664 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.256453 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-istio-data\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.256664 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.256487 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946a47c4-df05-4578-a544-05eb1e8672cd-istio-token\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.256664 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.256541 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946a47c4-df05-4578-a544-05eb1e8672cd-istiod-ca-cert\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.256664 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.256605 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-credential-socket\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.256664 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.256641 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqm7q\" (UniqueName: \"kubernetes.io/projected/946a47c4-df05-4578-a544-05eb1e8672cd-kube-api-access-zqm7q\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.358102 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.358063 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946a47c4-df05-4578-a544-05eb1e8672cd-istiod-ca-cert\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.358300 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.358118 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-credential-socket\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.358300 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.358140 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqm7q\" (UniqueName: \"kubernetes.io/projected/946a47c4-df05-4578-a544-05eb1e8672cd-kube-api-access-zqm7q\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.358300 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.358167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-workload-certs\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.358300 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.358195 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946a47c4-df05-4578-a544-05eb1e8672cd-istio-podinfo\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.358511 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.358364 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-workload-socket\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.358511 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.358422 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-istio-envoy\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.358511 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.358481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-istio-data\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.358511 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.358505 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946a47c4-df05-4578-a544-05eb1e8672cd-istio-token\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.358748 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.358587 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-credential-socket\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.358748 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.358639 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-workload-certs\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.358868 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.358770 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-workload-socket\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.359013 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.358995 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-istio-data\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.359082 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.359038 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946a47c4-df05-4578-a544-05eb1e8672cd-istiod-ca-cert\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.361279 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.361251 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-istio-envoy\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.361571 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.361551 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946a47c4-df05-4578-a544-05eb1e8672cd-istio-podinfo\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.368373 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.368301 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqm7q\" (UniqueName: \"kubernetes.io/projected/946a47c4-df05-4578-a544-05eb1e8672cd-kube-api-access-zqm7q\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.368747 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.368669 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946a47c4-df05-4578-a544-05eb1e8672cd-istio-token\") pod \"e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.430376 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.430337 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:22.575185 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.575155 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx"] Jul 06 20:02:22.577869 ip-10-0-140-94 kubenswrapper[2578]: W0706 20:02:22.577831 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946a47c4_df05_4578_a544_05eb1e8672cd.slice/crio-c568bc334048d43d65e07381bfce3ba32665d646c6bb738e21dfe5bab138b222 WatchSource:0}: Error finding container c568bc334048d43d65e07381bfce3ba32665d646c6bb738e21dfe5bab138b222: Status 404 returned error can't find the container with id c568bc334048d43d65e07381bfce3ba32665d646c6bb738e21dfe5bab138b222 Jul 06 20:02:22.580170 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.580136 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:02:22.580245 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.580204 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:02:22.580245 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:22.580235 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:02:23.406916 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:23.406878 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" event={"ID":"946a47c4-df05-4578-a544-05eb1e8672cd","Type":"ContainerStarted","Data":"33d391bd2b40b54b920c97810495c331607b8bf6673ed22a78a0afe8414b8580"} Jul 06 20:02:23.407321 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:23.406941 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" event={"ID":"946a47c4-df05-4578-a544-05eb1e8672cd","Type":"ContainerStarted","Data":"c568bc334048d43d65e07381bfce3ba32665d646c6bb738e21dfe5bab138b222"} Jul 06 20:02:23.430916 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:23.430881 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:23.435310 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:23.435255 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" podStartSLOduration=1.435237358 podStartE2EDuration="1.435237358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:02:23.432983174 +0000 UTC m=+2388.061937139" watchObservedRunningTime="2026-07-06 20:02:23.435237358 +0000 UTC m=+2388.064191323" Jul 06 20:02:23.436710 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:23.436677 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:24.412774 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:24.412738 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:24.413655 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:24.413637 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:26.434296 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.434261 2578 generic.go:356] "Generic (PLEG): container finished" podID="c1074b6d-ee33-49a9-a885-b836cbbfc19f" containerID="5c19e4c7dc146c4b8b74c18532aa681c0db8b6049c7cba516db39bf98e31023f" exitCode=0 Jul 06 20:02:26.434683 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.434333 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" event={"ID":"c1074b6d-ee33-49a9-a885-b836cbbfc19f","Type":"ContainerDied","Data":"5c19e4c7dc146c4b8b74c18532aa681c0db8b6049c7cba516db39bf98e31023f"} Jul 06 20:02:26.434683 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.434444 2578 scope.go:117] "RemoveContainer" containerID="29d0471223db1e472d46c6c7b6058ed9c1bce9b1a309041a075041f024ee451e" Jul 06 20:02:26.494239 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.494169 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:02:26.592677 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.592645 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-envoy\") pod \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " Jul 06 20:02:26.592677 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.592683 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-workload-certs\") pod \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " Jul 06 20:02:26.592937 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.592737 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-podinfo\") pod \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " Jul 06 20:02:26.592937 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.592767 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-workload-socket\") pod \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " Jul 06 20:02:26.592937 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.592825 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istiod-ca-cert\") pod \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " Jul 06 20:02:26.592937 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.592855 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-287px\" (UniqueName: \"kubernetes.io/projected/c1074b6d-ee33-49a9-a885-b836cbbfc19f-kube-api-access-287px\") pod \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " Jul 06 20:02:26.592937 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.592890 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-credential-socket\") pod \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " Jul 06 20:02:26.593191 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.592969 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-data\") pod \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " Jul 06 20:02:26.593191 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.593003 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-token\") pod \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\" (UID: \"c1074b6d-ee33-49a9-a885-b836cbbfc19f\") " Jul 06 20:02:26.593191 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.593030 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c1074b6d-ee33-49a9-a885-b836cbbfc19f" (UID: "c1074b6d-ee33-49a9-a885-b836cbbfc19f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:26.593358 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.593185 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c1074b6d-ee33-49a9-a885-b836cbbfc19f" (UID: "c1074b6d-ee33-49a9-a885-b836cbbfc19f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:26.593358 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.593281 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c1074b6d-ee33-49a9-a885-b836cbbfc19f" (UID: "c1074b6d-ee33-49a9-a885-b836cbbfc19f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:26.593358 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.593320 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-workload-certs\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:02:26.593358 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.593340 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-credential-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:02:26.593358 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.593319 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c1074b6d-ee33-49a9-a885-b836cbbfc19f" (UID: "c1074b6d-ee33-49a9-a885-b836cbbfc19f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:02:26.594063 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.594037 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-data" (OuterVolumeSpecName: "istio-data") pod "c1074b6d-ee33-49a9-a885-b836cbbfc19f" (UID: "c1074b6d-ee33-49a9-a885-b836cbbfc19f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:26.594957 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.594926 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c1074b6d-ee33-49a9-a885-b836cbbfc19f" (UID: "c1074b6d-ee33-49a9-a885-b836cbbfc19f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:26.595047 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.595033 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c1074b6d-ee33-49a9-a885-b836cbbfc19f" (UID: "c1074b6d-ee33-49a9-a885-b836cbbfc19f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:02:26.595196 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.595168 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-token" (OuterVolumeSpecName: "istio-token") pod "c1074b6d-ee33-49a9-a885-b836cbbfc19f" (UID: "c1074b6d-ee33-49a9-a885-b836cbbfc19f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:02:26.595768 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.595748 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1074b6d-ee33-49a9-a885-b836cbbfc19f-kube-api-access-287px" (OuterVolumeSpecName: "kube-api-access-287px") pod "c1074b6d-ee33-49a9-a885-b836cbbfc19f" (UID: "c1074b6d-ee33-49a9-a885-b836cbbfc19f"). InnerVolumeSpecName "kube-api-access-287px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:02:26.693851 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.693814 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istiod-ca-cert\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:02:26.693851 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.693846 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-287px\" (UniqueName: \"kubernetes.io/projected/c1074b6d-ee33-49a9-a885-b836cbbfc19f-kube-api-access-287px\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:02:26.693851 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.693859 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-data\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:02:26.694097 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.693868 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-token\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:02:26.694097 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.693877 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-envoy\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:02:26.694097 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.693884 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1074b6d-ee33-49a9-a885-b836cbbfc19f-istio-podinfo\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:02:26.694097 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:26.693891 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1074b6d-ee33-49a9-a885-b836cbbfc19f-workload-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:02:27.441055 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:27.441027 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" Jul 06 20:02:27.441500 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:27.441077 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch" event={"ID":"c1074b6d-ee33-49a9-a885-b836cbbfc19f","Type":"ContainerDied","Data":"27bcee163e01a78f5cf6d8f9b46b60d9ddadd1d08a0c503a287e542aab7c5c6c"} Jul 06 20:02:27.441500 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:27.441137 2578 scope.go:117] "RemoveContainer" containerID="5c19e4c7dc146c4b8b74c18532aa681c0db8b6049c7cba516db39bf98e31023f" Jul 06 20:02:27.477798 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:27.477771 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch"] Jul 06 20:02:27.486372 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:27.486348 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a82399-openshift-default-74c459cf65-292ch"] Jul 06 20:02:27.876203 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:27.876163 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1074b6d-ee33-49a9-a885-b836cbbfc19f" path="/var/lib/kubelet/pods/c1074b6d-ee33-49a9-a885-b836cbbfc19f/volumes" Jul 06 20:02:30.560422 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.560389 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd"] Jul 06 20:02:30.560851 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.560837 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1074b6d-ee33-49a9-a885-b836cbbfc19f" containerName="istio-proxy" Jul 06 20:02:30.560851 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.560851 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1074b6d-ee33-49a9-a885-b836cbbfc19f" containerName="istio-proxy" Jul 06 20:02:30.560941 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.560858 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1074b6d-ee33-49a9-a885-b836cbbfc19f" containerName="istio-proxy" Jul 06 20:02:30.560941 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.560863 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1074b6d-ee33-49a9-a885-b836cbbfc19f" containerName="istio-proxy" Jul 06 20:02:30.561003 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.560944 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1074b6d-ee33-49a9-a885-b836cbbfc19f" containerName="istio-proxy" Jul 06 20:02:30.565482 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.565467 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.569042 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.569001 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-f7a150-openshift-default-dockercfg-g5m5v\"" Jul 06 20:02:30.589152 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.589117 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd"] Jul 06 20:02:30.632152 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.632113 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istiod-ca-cert\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.632352 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.632162 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-data\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.632352 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.632228 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-workload-socket\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.632352 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.632248 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-token\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.632352 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.632323 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-envoy\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.632567 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.632418 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-workload-certs\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.632567 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.632449 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gkg9\" (UniqueName: \"kubernetes.io/projected/5d51bb6a-0881-437a-8bd3-ecc6b7886135-kube-api-access-5gkg9\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.632567 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.632492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-credential-socket\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.632567 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.632524 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-podinfo\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.733592 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.733556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-workload-certs\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.733592 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.733595 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gkg9\" (UniqueName: \"kubernetes.io/projected/5d51bb6a-0881-437a-8bd3-ecc6b7886135-kube-api-access-5gkg9\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.733868 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.733742 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-credential-socket\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.733868 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.733792 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-podinfo\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.733987 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.733881 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istiod-ca-cert\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.734455 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.733916 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-data\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.734579 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.734515 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-workload-socket\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.734579 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.734542 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-token\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.734579 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.734163 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-credential-socket\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.734788 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.734582 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istiod-ca-cert\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.734788 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.734028 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-workload-certs\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.734788 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.734247 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-data\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.734788 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.734574 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-envoy\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.734999 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.734851 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-workload-socket\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.736673 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.736644 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-podinfo\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.736793 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.736748 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-envoy\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.746930 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.746899 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gkg9\" (UniqueName: \"kubernetes.io/projected/5d51bb6a-0881-437a-8bd3-ecc6b7886135-kube-api-access-5gkg9\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.748031 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.748002 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-token\") pod \"e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:30.878620 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:30.878523 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:31.033342 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:31.033312 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd"] Jul 06 20:02:31.035482 ip-10-0-140-94 kubenswrapper[2578]: W0706 20:02:31.035447 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d51bb6a_0881_437a_8bd3_ecc6b7886135.slice/crio-eafe8e45cbf506bae43e87f3631e15441a8737f8ae0795a5fc338aee4e356f2d WatchSource:0}: Error finding container eafe8e45cbf506bae43e87f3631e15441a8737f8ae0795a5fc338aee4e356f2d: Status 404 returned error can't find the container with id eafe8e45cbf506bae43e87f3631e15441a8737f8ae0795a5fc338aee4e356f2d Jul 06 20:02:31.037672 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:31.037639 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:02:31.037800 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:31.037772 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:02:31.037861 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:31.037822 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:02:31.464181 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:31.464143 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" event={"ID":"5d51bb6a-0881-437a-8bd3-ecc6b7886135","Type":"ContainerStarted","Data":"18f8486a1bf81ffb86ee4d0c4a86f33dd32f2121e44d19c73a9f6bcc9e81df0c"} Jul 06 20:02:31.464401 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:31.464194 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" event={"ID":"5d51bb6a-0881-437a-8bd3-ecc6b7886135","Type":"ContainerStarted","Data":"eafe8e45cbf506bae43e87f3631e15441a8737f8ae0795a5fc338aee4e356f2d"} Jul 06 20:02:31.488085 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:31.488033 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" podStartSLOduration=1.488017811 podStartE2EDuration="1.488017811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:02:31.485988027 +0000 UTC m=+2396.114941992" watchObservedRunningTime="2026-07-06 20:02:31.488017811 +0000 UTC m=+2396.116971849" Jul 06 20:02:31.878777 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:31.878748 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:32.884022 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:32.883990 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:33.476061 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:33.476031 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:33.476968 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:33.476949 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:36.057092 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:36.057062 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 20:02:36.065302 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:36.065281 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 20:02:42.526648 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:42.526617 2578 generic.go:356] "Generic (PLEG): container finished" podID="946a47c4-df05-4578-a544-05eb1e8672cd" containerID="33d391bd2b40b54b920c97810495c331607b8bf6673ed22a78a0afe8414b8580" exitCode=0 Jul 06 20:02:42.527123 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:42.526683 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" event={"ID":"946a47c4-df05-4578-a544-05eb1e8672cd","Type":"ContainerDied","Data":"33d391bd2b40b54b920c97810495c331607b8bf6673ed22a78a0afe8414b8580"} Jul 06 20:02:42.532574 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:42.532145 2578 scope.go:117] "RemoveContainer" containerID="33d391bd2b40b54b920c97810495c331607b8bf6673ed22a78a0afe8414b8580" Jul 06 20:02:42.534959 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:42.533638 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:02:42.534959 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:42.533755 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:02:42.534959 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:42.533804 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:02:43.537643 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:43.537596 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" event={"ID":"946a47c4-df05-4578-a544-05eb1e8672cd","Type":"ContainerStarted","Data":"2cfe3b8ca4163c5110ed19233d631d65bae795ca5d60e3f0f608749808daee06"} Jul 06 20:02:44.431640 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:44.431594 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:44.431886 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:44.431776 2578 patch_prober.go:28] interesting pod/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 06 20:02:44.431886 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:44.431835 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" podUID="946a47c4-df05-4578-a544-05eb1e8672cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 06 20:02:45.431218 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:45.431169 2578 patch_prober.go:28] interesting pod/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 06 20:02:45.431746 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:45.431260 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" podUID="946a47c4-df05-4578-a544-05eb1e8672cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 06 20:02:46.431431 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:46.431291 2578 patch_prober.go:28] interesting pod/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 06 20:02:46.431431 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:46.431375 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" podUID="946a47c4-df05-4578-a544-05eb1e8672cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 06 20:02:47.431988 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:47.431855 2578 patch_prober.go:28] interesting pod/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 06 20:02:47.431988 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:47.431928 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" podUID="946a47c4-df05-4578-a544-05eb1e8672cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 06 20:02:48.434318 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:48.434284 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:48.434735 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:48.434560 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:48.435191 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:48.435170 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:02:49.071861 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:49.071824 2578 patch_prober.go:28] interesting pod/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": read tcp 10.132.0.2:44334->10.132.0.56:15021: read: connection reset by peer" start-of-body= Jul 06 20:02:49.072041 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:49.071897 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" podUID="5d51bb6a-0881-437a-8bd3-ecc6b7886135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": read tcp 10.132.0.2:44334->10.132.0.56:15021: read: connection reset by peer" Jul 06 20:02:49.578587 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:49.578561 2578 generic.go:356] "Generic (PLEG): container finished" podID="5d51bb6a-0881-437a-8bd3-ecc6b7886135" containerID="18f8486a1bf81ffb86ee4d0c4a86f33dd32f2121e44d19c73a9f6bcc9e81df0c" exitCode=0 Jul 06 20:02:49.579059 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:49.578643 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" event={"ID":"5d51bb6a-0881-437a-8bd3-ecc6b7886135","Type":"ContainerDied","Data":"18f8486a1bf81ffb86ee4d0c4a86f33dd32f2121e44d19c73a9f6bcc9e81df0c"} Jul 06 20:02:49.579247 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:49.579231 2578 scope.go:117] "RemoveContainer" containerID="18f8486a1bf81ffb86ee4d0c4a86f33dd32f2121e44d19c73a9f6bcc9e81df0c" Jul 06 20:02:49.580189 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:49.580163 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:02:49.580245 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:49.580228 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:02:49.580280 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:49.580259 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 20:02:50.588281 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:50.588246 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" event={"ID":"5d51bb6a-0881-437a-8bd3-ecc6b7886135","Type":"ContainerStarted","Data":"58b17efe80a2e69cf3407ee50e3027385051200e2f80c930ea5a36f1caddfc8d"} Jul 06 20:02:50.879595 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:50.879499 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:50.879788 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:50.879747 2578 patch_prober.go:28] interesting pod/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 06 20:02:50.879834 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:50.879794 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" podUID="5d51bb6a-0881-437a-8bd3-ecc6b7886135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 06 20:02:51.879834 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:51.879801 2578 patch_prober.go:28] interesting pod/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 06 20:02:51.880239 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:51.879851 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" podUID="5d51bb6a-0881-437a-8bd3-ecc6b7886135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 06 20:02:52.879283 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:52.879249 2578 patch_prober.go:28] interesting pod/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 06 20:02:52.879472 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:52.879305 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" podUID="5d51bb6a-0881-437a-8bd3-ecc6b7886135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 06 20:02:53.879209 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:53.879174 2578 patch_prober.go:28] interesting pod/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 06 20:02:53.879596 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:53.879225 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" podUID="5d51bb6a-0881-437a-8bd3-ecc6b7886135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 06 20:02:54.883208 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:54.883174 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:54.883680 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:54.883438 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:02:54.884232 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:02:54.884216 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:07:36.091266 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:07:36.091234 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 20:07:36.101267 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:07:36.101242 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 20:09:00.540935 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:00.540845 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx"] Jul 06 20:09:00.541474 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:00.541090 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" podUID="946a47c4-df05-4578-a544-05eb1e8672cd" containerName="istio-proxy" containerID="cri-o://2cfe3b8ca4163c5110ed19233d631d65bae795ca5d60e3f0f608749808daee06" gracePeriod=30 Jul 06 20:09:03.437327 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:03.437287 2578 patch_prober.go:28] interesting pod/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 20:09:03.437751 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:03.437358 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" podUID="946a47c4-df05-4578-a544-05eb1e8672cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:09:05.791883 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.791850 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:09:05.897432 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.897340 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-credential-socket\") pod \"946a47c4-df05-4578-a544-05eb1e8672cd\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " Jul 06 20:09:05.897577 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.897451 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqm7q\" (UniqueName: \"kubernetes.io/projected/946a47c4-df05-4578-a544-05eb1e8672cd-kube-api-access-zqm7q\") pod \"946a47c4-df05-4578-a544-05eb1e8672cd\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " Jul 06 20:09:05.897577 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.897474 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-istio-envoy\") pod \"946a47c4-df05-4578-a544-05eb1e8672cd\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " Jul 06 20:09:05.897577 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.897491 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946a47c4-df05-4578-a544-05eb1e8672cd-istiod-ca-cert\") pod \"946a47c4-df05-4578-a544-05eb1e8672cd\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " Jul 06 20:09:05.897577 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.897535 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946a47c4-df05-4578-a544-05eb1e8672cd-istio-podinfo\") pod \"946a47c4-df05-4578-a544-05eb1e8672cd\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " Jul 06 20:09:05.897820 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.897617 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946a47c4-df05-4578-a544-05eb1e8672cd-istio-token\") pod \"946a47c4-df05-4578-a544-05eb1e8672cd\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " Jul 06 20:09:05.897820 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.897639 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "946a47c4-df05-4578-a544-05eb1e8672cd" (UID: "946a47c4-df05-4578-a544-05eb1e8672cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:09:05.897820 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.897728 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-istio-data\") pod \"946a47c4-df05-4578-a544-05eb1e8672cd\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " Jul 06 20:09:05.897820 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.897765 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-workload-socket\") pod \"946a47c4-df05-4578-a544-05eb1e8672cd\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " Jul 06 20:09:05.898033 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.897823 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-workload-certs\") pod \"946a47c4-df05-4578-a544-05eb1e8672cd\" (UID: \"946a47c4-df05-4578-a544-05eb1e8672cd\") " Jul 06 20:09:05.898101 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.897961 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946a47c4-df05-4578-a544-05eb1e8672cd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "946a47c4-df05-4578-a544-05eb1e8672cd" (UID: "946a47c4-df05-4578-a544-05eb1e8672cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:09:05.898233 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.898211 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-credential-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:09:05.898314 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.898239 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946a47c4-df05-4578-a544-05eb1e8672cd-istiod-ca-cert\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:09:05.898388 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.898364 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "946a47c4-df05-4578-a544-05eb1e8672cd" (UID: "946a47c4-df05-4578-a544-05eb1e8672cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:09:05.898554 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.898535 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "946a47c4-df05-4578-a544-05eb1e8672cd" (UID: "946a47c4-df05-4578-a544-05eb1e8672cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:09:05.898657 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.898628 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-istio-data" (OuterVolumeSpecName: "istio-data") pod "946a47c4-df05-4578-a544-05eb1e8672cd" (UID: "946a47c4-df05-4578-a544-05eb1e8672cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:09:05.899952 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.899923 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946a47c4-df05-4578-a544-05eb1e8672cd-istio-token" (OuterVolumeSpecName: "istio-token") pod "946a47c4-df05-4578-a544-05eb1e8672cd" (UID: "946a47c4-df05-4578-a544-05eb1e8672cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:09:05.900064 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.900031 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946a47c4-df05-4578-a544-05eb1e8672cd-kube-api-access-zqm7q" (OuterVolumeSpecName: "kube-api-access-zqm7q") pod "946a47c4-df05-4578-a544-05eb1e8672cd" (UID: "946a47c4-df05-4578-a544-05eb1e8672cd"). InnerVolumeSpecName "kube-api-access-zqm7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:09:05.900253 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.900229 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "946a47c4-df05-4578-a544-05eb1e8672cd" (UID: "946a47c4-df05-4578-a544-05eb1e8672cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:09:05.900359 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.900235 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/946a47c4-df05-4578-a544-05eb1e8672cd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "946a47c4-df05-4578-a544-05eb1e8672cd" (UID: "946a47c4-df05-4578-a544-05eb1e8672cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:09:05.999141 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.999107 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-istio-data\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:09:05.999141 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.999138 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-workload-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:09:05.999141 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.999147 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-workload-certs\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:09:05.999376 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.999157 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqm7q\" (UniqueName: \"kubernetes.io/projected/946a47c4-df05-4578-a544-05eb1e8672cd-kube-api-access-zqm7q\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:09:05.999376 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.999166 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946a47c4-df05-4578-a544-05eb1e8672cd-istio-envoy\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:09:05.999376 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.999174 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946a47c4-df05-4578-a544-05eb1e8672cd-istio-podinfo\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:09:05.999376 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:05.999181 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946a47c4-df05-4578-a544-05eb1e8672cd-istio-token\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:09:06.619436 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:06.619399 2578 generic.go:356] "Generic (PLEG): container finished" podID="946a47c4-df05-4578-a544-05eb1e8672cd" containerID="2cfe3b8ca4163c5110ed19233d631d65bae795ca5d60e3f0f608749808daee06" exitCode=0 Jul 06 20:09:06.619610 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:06.619466 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" event={"ID":"946a47c4-df05-4578-a544-05eb1e8672cd","Type":"ContainerDied","Data":"2cfe3b8ca4163c5110ed19233d631d65bae795ca5d60e3f0f608749808daee06"} Jul 06 20:09:06.619610 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:06.619498 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" Jul 06 20:09:06.619610 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:06.619518 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx" event={"ID":"946a47c4-df05-4578-a544-05eb1e8672cd","Type":"ContainerDied","Data":"c568bc334048d43d65e07381bfce3ba32665d646c6bb738e21dfe5bab138b222"} Jul 06 20:09:06.619610 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:06.619548 2578 scope.go:117] "RemoveContainer" containerID="2cfe3b8ca4163c5110ed19233d631d65bae795ca5d60e3f0f608749808daee06" Jul 06 20:09:06.631134 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:06.631103 2578 scope.go:117] "RemoveContainer" containerID="33d391bd2b40b54b920c97810495c331607b8bf6673ed22a78a0afe8414b8580" Jul 06 20:09:06.641310 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:06.641290 2578 scope.go:117] "RemoveContainer" containerID="2cfe3b8ca4163c5110ed19233d631d65bae795ca5d60e3f0f608749808daee06" Jul 06 20:09:06.641613 ip-10-0-140-94 kubenswrapper[2578]: E0706 20:09:06.641591 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cfe3b8ca4163c5110ed19233d631d65bae795ca5d60e3f0f608749808daee06\": container with ID starting with 2cfe3b8ca4163c5110ed19233d631d65bae795ca5d60e3f0f608749808daee06 not found: ID does not exist" containerID="2cfe3b8ca4163c5110ed19233d631d65bae795ca5d60e3f0f608749808daee06" Jul 06 20:09:06.641754 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:06.641619 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cfe3b8ca4163c5110ed19233d631d65bae795ca5d60e3f0f608749808daee06"} err="failed to get container status \"2cfe3b8ca4163c5110ed19233d631d65bae795ca5d60e3f0f608749808daee06\": rpc error: code = NotFound desc = could not find container \"2cfe3b8ca4163c5110ed19233d631d65bae795ca5d60e3f0f608749808daee06\": container with ID starting with 2cfe3b8ca4163c5110ed19233d631d65bae795ca5d60e3f0f608749808daee06 not found: ID does not exist" Jul 06 20:09:06.641754 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:06.641642 2578 scope.go:117] "RemoveContainer" containerID="33d391bd2b40b54b920c97810495c331607b8bf6673ed22a78a0afe8414b8580" Jul 06 20:09:06.641916 ip-10-0-140-94 kubenswrapper[2578]: E0706 20:09:06.641891 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33d391bd2b40b54b920c97810495c331607b8bf6673ed22a78a0afe8414b8580\": container with ID starting with 33d391bd2b40b54b920c97810495c331607b8bf6673ed22a78a0afe8414b8580 not found: ID does not exist" containerID="33d391bd2b40b54b920c97810495c331607b8bf6673ed22a78a0afe8414b8580" Jul 06 20:09:06.642001 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:06.641917 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33d391bd2b40b54b920c97810495c331607b8bf6673ed22a78a0afe8414b8580"} err="failed to get container status \"33d391bd2b40b54b920c97810495c331607b8bf6673ed22a78a0afe8414b8580\": rpc error: code = NotFound desc = could not find container \"33d391bd2b40b54b920c97810495c331607b8bf6673ed22a78a0afe8414b8580\": container with ID starting with 33d391bd2b40b54b920c97810495c331607b8bf6673ed22a78a0afe8414b8580 not found: ID does not exist" Jul 06 20:09:06.667148 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:06.667118 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx"] Jul 06 20:09:06.681288 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:06.681263 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-c21a78-openshift-default-96895bfd7-7w7rx"] Jul 06 20:09:07.876675 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:07.876639 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="946a47c4-df05-4578-a544-05eb1e8672cd" path="/var/lib/kubelet/pods/946a47c4-df05-4578-a544-05eb1e8672cd/volumes" Jul 06 20:09:09.388230 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:09.388196 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd"] Jul 06 20:09:09.388645 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:09.388431 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" podUID="5d51bb6a-0881-437a-8bd3-ecc6b7886135" containerName="istio-proxy" containerID="cri-o://58b17efe80a2e69cf3407ee50e3027385051200e2f80c930ea5a36f1caddfc8d" gracePeriod=30 Jul 06 20:09:09.886065 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:09.886027 2578 patch_prober.go:28] interesting pod/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 20:09:09.886223 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:09.886089 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" podUID="5d51bb6a-0881-437a-8bd3-ecc6b7886135" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:09:14.632069 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.632016 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pv5d5_40393be6-bd01-4617-ace4-5cbdfcd186fd/manager/3.log" Jul 06 20:09:14.656957 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.656898 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:09:14.668232 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.668204 2578 generic.go:356] "Generic (PLEG): container finished" podID="5d51bb6a-0881-437a-8bd3-ecc6b7886135" containerID="58b17efe80a2e69cf3407ee50e3027385051200e2f80c930ea5a36f1caddfc8d" exitCode=0 Jul 06 20:09:14.668421 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.668246 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" event={"ID":"5d51bb6a-0881-437a-8bd3-ecc6b7886135","Type":"ContainerDied","Data":"58b17efe80a2e69cf3407ee50e3027385051200e2f80c930ea5a36f1caddfc8d"} Jul 06 20:09:14.668421 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.668253 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" Jul 06 20:09:14.668421 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.668315 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd" event={"ID":"5d51bb6a-0881-437a-8bd3-ecc6b7886135","Type":"ContainerDied","Data":"eafe8e45cbf506bae43e87f3631e15441a8737f8ae0795a5fc338aee4e356f2d"} Jul 06 20:09:14.668421 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.668336 2578 scope.go:117] "RemoveContainer" containerID="58b17efe80a2e69cf3407ee50e3027385051200e2f80c930ea5a36f1caddfc8d" Jul 06 20:09:14.677718 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.677673 2578 scope.go:117] "RemoveContainer" containerID="18f8486a1bf81ffb86ee4d0c4a86f33dd32f2121e44d19c73a9f6bcc9e81df0c" Jul 06 20:09:14.688016 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.687995 2578 scope.go:117] "RemoveContainer" containerID="58b17efe80a2e69cf3407ee50e3027385051200e2f80c930ea5a36f1caddfc8d" Jul 06 20:09:14.688366 ip-10-0-140-94 kubenswrapper[2578]: E0706 20:09:14.688342 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b17efe80a2e69cf3407ee50e3027385051200e2f80c930ea5a36f1caddfc8d\": container with ID starting with 58b17efe80a2e69cf3407ee50e3027385051200e2f80c930ea5a36f1caddfc8d not found: ID does not exist" containerID="58b17efe80a2e69cf3407ee50e3027385051200e2f80c930ea5a36f1caddfc8d" Jul 06 20:09:14.688480 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.688378 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b17efe80a2e69cf3407ee50e3027385051200e2f80c930ea5a36f1caddfc8d"} err="failed to get container status \"58b17efe80a2e69cf3407ee50e3027385051200e2f80c930ea5a36f1caddfc8d\": rpc error: code = NotFound desc = could not find container \"58b17efe80a2e69cf3407ee50e3027385051200e2f80c930ea5a36f1caddfc8d\": container with ID starting with 58b17efe80a2e69cf3407ee50e3027385051200e2f80c930ea5a36f1caddfc8d not found: ID does not exist" Jul 06 20:09:14.688480 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.688403 2578 scope.go:117] "RemoveContainer" containerID="18f8486a1bf81ffb86ee4d0c4a86f33dd32f2121e44d19c73a9f6bcc9e81df0c" Jul 06 20:09:14.688710 ip-10-0-140-94 kubenswrapper[2578]: E0706 20:09:14.688674 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f8486a1bf81ffb86ee4d0c4a86f33dd32f2121e44d19c73a9f6bcc9e81df0c\": container with ID starting with 18f8486a1bf81ffb86ee4d0c4a86f33dd32f2121e44d19c73a9f6bcc9e81df0c not found: ID does not exist" containerID="18f8486a1bf81ffb86ee4d0c4a86f33dd32f2121e44d19c73a9f6bcc9e81df0c" Jul 06 20:09:14.688780 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.688723 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f8486a1bf81ffb86ee4d0c4a86f33dd32f2121e44d19c73a9f6bcc9e81df0c"} err="failed to get container status \"18f8486a1bf81ffb86ee4d0c4a86f33dd32f2121e44d19c73a9f6bcc9e81df0c\": rpc error: code = NotFound desc = could not find container \"18f8486a1bf81ffb86ee4d0c4a86f33dd32f2121e44d19c73a9f6bcc9e81df0c\": container with ID starting with 18f8486a1bf81ffb86ee4d0c4a86f33dd32f2121e44d19c73a9f6bcc9e81df0c not found: ID does not exist" Jul 06 20:09:14.767290 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.767254 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hrvh5_f55184b1-bd4b-4da5-9f7a-d6099ed678ab/manager/3.log" Jul 06 20:09:14.790922 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.790894 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-envoy\") pod \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " Jul 06 20:09:14.791112 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.790939 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gkg9\" (UniqueName: \"kubernetes.io/projected/5d51bb6a-0881-437a-8bd3-ecc6b7886135-kube-api-access-5gkg9\") pod \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " Jul 06 20:09:14.791112 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.790973 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-podinfo\") pod \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " Jul 06 20:09:14.791112 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.791003 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-credential-socket\") pod \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " Jul 06 20:09:14.791112 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.791040 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istiod-ca-cert\") pod \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " Jul 06 20:09:14.791112 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.791083 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-data\") pod \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " Jul 06 20:09:14.791388 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.791161 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-token\") pod \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " Jul 06 20:09:14.791388 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.791193 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-workload-socket\") pod \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " Jul 06 20:09:14.791388 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.791216 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-workload-certs\") pod \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\" (UID: \"5d51bb6a-0881-437a-8bd3-ecc6b7886135\") " Jul 06 20:09:14.791553 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.791425 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5d51bb6a-0881-437a-8bd3-ecc6b7886135" (UID: "5d51bb6a-0881-437a-8bd3-ecc6b7886135"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:09:14.791616 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.791594 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-credential-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:09:14.791669 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.791624 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5d51bb6a-0881-437a-8bd3-ecc6b7886135" (UID: "5d51bb6a-0881-437a-8bd3-ecc6b7886135"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:09:14.791754 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.791680 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5d51bb6a-0881-437a-8bd3-ecc6b7886135" (UID: "5d51bb6a-0881-437a-8bd3-ecc6b7886135"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:09:14.791838 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.791817 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5d51bb6a-0881-437a-8bd3-ecc6b7886135" (UID: "5d51bb6a-0881-437a-8bd3-ecc6b7886135"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:09:14.792045 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.792014 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-data" (OuterVolumeSpecName: "istio-data") pod "5d51bb6a-0881-437a-8bd3-ecc6b7886135" (UID: "5d51bb6a-0881-437a-8bd3-ecc6b7886135"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:09:14.793377 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.793351 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5d51bb6a-0881-437a-8bd3-ecc6b7886135" (UID: "5d51bb6a-0881-437a-8bd3-ecc6b7886135"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:09:14.793473 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.793349 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-token" (OuterVolumeSpecName: "istio-token") pod "5d51bb6a-0881-437a-8bd3-ecc6b7886135" (UID: "5d51bb6a-0881-437a-8bd3-ecc6b7886135"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:09:14.793473 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.793406 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d51bb6a-0881-437a-8bd3-ecc6b7886135-kube-api-access-5gkg9" (OuterVolumeSpecName: "kube-api-access-5gkg9") pod "5d51bb6a-0881-437a-8bd3-ecc6b7886135" (UID: "5d51bb6a-0881-437a-8bd3-ecc6b7886135"). InnerVolumeSpecName "kube-api-access-5gkg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:09:14.793662 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.793642 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5d51bb6a-0881-437a-8bd3-ecc6b7886135" (UID: "5d51bb6a-0881-437a-8bd3-ecc6b7886135"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:09:14.892575 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.892535 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-token\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:09:14.892575 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.892568 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-workload-socket\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:09:14.892575 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.892577 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-workload-certs\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:09:14.892841 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.892587 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-envoy\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:09:14.892841 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.892596 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gkg9\" (UniqueName: \"kubernetes.io/projected/5d51bb6a-0881-437a-8bd3-ecc6b7886135-kube-api-access-5gkg9\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:09:14.892841 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.892604 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-podinfo\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:09:14.892841 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.892611 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istiod-ca-cert\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:09:14.892841 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.892619 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d51bb6a-0881-437a-8bd3-ecc6b7886135-istio-data\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\"" Jul 06 20:09:14.999445 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:14.999408 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd"] Jul 06 20:09:15.007895 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:15.007855 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-f7a150-openshift-default-6c8fddd5ff-xhqxd"] Jul 06 20:09:15.016347 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:15.016324 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722785-sbhn6_32fd3226-5ae9-4c8a-b8a7-6fec412df49c/cleanup/0.log" Jul 06 20:09:15.527937 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:15.527907 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-kkjcw_85caf392-2962-4d96-8ae0-59dbf3366abe/manager/4.log" Jul 06 20:09:15.662780 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:15.662750 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7865995f8c-mwbc8_f1f2d36b-4239-45fa-a6d7-85c316bd426e/copy-manifests/0.log" Jul 06 20:09:15.675656 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:15.675631 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7865995f8c-mwbc8_f1f2d36b-4239-45fa-a6d7-85c316bd426e/manager/2.log" Jul 06 20:09:15.876107 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:15.876026 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d51bb6a-0881-437a-8bd3-ecc6b7886135" path="/var/lib/kubelet/pods/5d51bb6a-0881-437a-8bd3-ecc6b7886135/volumes" Jul 06 20:09:16.153827 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:16.153752 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-xc6pb_c9acfc53-44f0-4ccd-9768-7d17fb00394d/postgres/0.log" Jul 06 20:09:17.848762 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:17.848733 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-ndlnz_7f55d1c1-4236-45e8-812a-68f2b9015ff6/manager/3.log" Jul 06 20:09:17.962640 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:17.962610 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-6br2w_18778df4-9ca2-41fe-a983-e69f2d564dd2/manager/3.log" Jul 06 20:09:18.085956 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:18.085931 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-lrmt9_d343a282-5e48-4a26-acb0-e1818285d7eb/kuadrant-console-plugin/0.log" Jul 06 20:09:18.203156 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:18.203082 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-84jdz_5bd6e3a8-f67a-4255-b7c9-13958774c94c/registry-server/0.log" Jul 06 20:09:18.579815 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:18.579783 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-4lrbz_fc0a6e65-42be-43eb-8a55-363e166ecc16/manager/3.log" Jul 06 20:09:18.931085 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:18.930995 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6786q4mq_373da31a-f406-40cb-b233-e4e7f2cbf777/istio-proxy/0.log" Jul 06 20:09:19.164753 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:19.164723 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67db589d55-p5z7q_2342b288-6681-49f1-85ca-261c3fe2e33d/kube-auth-proxy/0.log" Jul 06 20:09:19.389514 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:19.389488 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nxb4p_639168ba-36ec-47a3-a023-6135f27dd8d9/istio-proxy/1.log" Jul 06 20:09:19.631623 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:19.631590 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-45k9w_2e41d8b1-2901-40df-bcc1-9314078eebae/payload-processing/0.log" Jul 06 20:09:19.743598 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:19.743522 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68d9d48ccc-kk7pz_cc56f897-5d59-4451-a517-5bfea063ce0c/router/0.log" Jul 06 20:09:20.451987 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:20.451952 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbn9vrl_9122cc34-84f5-4484-8e9a-981cdf612efb/main/0.log" Jul 06 20:09:28.064472 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:28.064441 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kpsb4_a6d9fe54-0654-4f9d-bb79-0fbeec14c98f/global-pull-secret-syncer/0.log" Jul 06 20:09:28.231044 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:28.231011 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wcsxd_63219616-9c15-4abc-90ee-42160928e479/konnectivity-agent/0.log" Jul 06 20:09:28.334542 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:28.334458 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-94.ec2.internal_aca81e81575313b78c60adfe6a1aa8fd/haproxy/0.log" Jul 06 20:09:33.563105 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:33.563076 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-ndlnz_7f55d1c1-4236-45e8-812a-68f2b9015ff6/manager/2.log" Jul 06 20:09:33.563547 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:33.563529 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-ndlnz_7f55d1c1-4236-45e8-812a-68f2b9015ff6/manager/3.log" Jul 06 20:09:33.593936 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:33.593904 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-6br2w_18778df4-9ca2-41fe-a983-e69f2d564dd2/manager/3.log" Jul 06 20:09:33.594130 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:33.594083 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-6br2w_18778df4-9ca2-41fe-a983-e69f2d564dd2/manager/2.log" Jul 06 20:09:33.621592 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:33.621567 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-lrmt9_d343a282-5e48-4a26-acb0-e1818285d7eb/kuadrant-console-plugin/0.log" Jul 06 20:09:33.675554 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:33.675521 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-84jdz_5bd6e3a8-f67a-4255-b7c9-13958774c94c/registry-server/0.log" Jul 06 20:09:33.953089 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:33.952994 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-4lrbz_fc0a6e65-42be-43eb-8a55-363e166ecc16/manager/2.log" Jul 06 20:09:33.977077 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:33.977042 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-4lrbz_fc0a6e65-42be-43eb-8a55-363e166ecc16/manager/3.log" Jul 06 20:09:36.261806 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:36.261762 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-z8zqc_8aa685e5-fc07-4d7b-bb90-506c9c9f34be/cluster-monitoring-operator/0.log" Jul 06 20:09:36.805855 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:36.805826 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dsgzx_7611af4b-652f-4386-ae4a-39d0d802e146/node-exporter/0.log" Jul 06 20:09:36.853241 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:36.853213 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dsgzx_7611af4b-652f-4386-ae4a-39d0d802e146/kube-rbac-proxy/0.log" Jul 06 20:09:36.889799 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:36.889773 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dsgzx_7611af4b-652f-4386-ae4a-39d0d802e146/init-textfile/0.log" Jul 06 20:09:37.551337 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.551286 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-hstnz_5bbb14fe-9b7b-45d8-9401-3ff9468879d0/prometheus-operator-admission-webhook/0.log" Jul 06 20:09:37.924130 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.924033 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm"] Jul 06 20:09:37.924449 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.924436 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="946a47c4-df05-4578-a544-05eb1e8672cd" containerName="istio-proxy" Jul 06 20:09:37.924497 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.924450 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="946a47c4-df05-4578-a544-05eb1e8672cd" containerName="istio-proxy" Jul 06 20:09:37.924497 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.924460 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d51bb6a-0881-437a-8bd3-ecc6b7886135" containerName="istio-proxy" Jul 06 20:09:37.924497 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.924466 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d51bb6a-0881-437a-8bd3-ecc6b7886135" containerName="istio-proxy" Jul 06 20:09:37.924497 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.924483 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="946a47c4-df05-4578-a544-05eb1e8672cd" containerName="istio-proxy" Jul 06 20:09:37.924497 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.924489 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="946a47c4-df05-4578-a544-05eb1e8672cd" containerName="istio-proxy" Jul 06 20:09:37.924497 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.924497 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d51bb6a-0881-437a-8bd3-ecc6b7886135" containerName="istio-proxy" Jul 06 20:09:37.924723 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.924502 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d51bb6a-0881-437a-8bd3-ecc6b7886135" containerName="istio-proxy" Jul 06 20:09:37.924723 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.924572 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="946a47c4-df05-4578-a544-05eb1e8672cd" containerName="istio-proxy" Jul 06 20:09:37.924723 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.924578 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d51bb6a-0881-437a-8bd3-ecc6b7886135" containerName="istio-proxy" Jul 06 20:09:37.924723 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.924584 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="946a47c4-df05-4578-a544-05eb1e8672cd" containerName="istio-proxy" Jul 06 20:09:37.924723 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.924590 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1074b6d-ee33-49a9-a885-b836cbbfc19f" containerName="istio-proxy" Jul 06 20:09:37.924723 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.924596 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d51bb6a-0881-437a-8bd3-ecc6b7886135" containerName="istio-proxy" Jul 06 20:09:37.929435 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.929416 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" Jul 06 20:09:37.932566 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.932546 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tfs5f\"/\"kube-root-ca.crt\"" Jul 06 20:09:37.933263 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.933245 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tfs5f\"/\"default-dockercfg-2txjc\"" Jul 06 20:09:37.933338 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.933316 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tfs5f\"/\"openshift-service-ca.crt\"" Jul 06 20:09:37.960838 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:37.960803 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm"] Jul 06 20:09:38.108351 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.108318 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a82f624-b48c-434f-bd7c-9fc1e742b8af-podres\") pod \"perf-node-gather-daemonset-mgkmm\" (UID: \"1a82f624-b48c-434f-bd7c-9fc1e742b8af\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" Jul 06 20:09:38.108555 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.108367 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a82f624-b48c-434f-bd7c-9fc1e742b8af-sys\") pod \"perf-node-gather-daemonset-mgkmm\" (UID: \"1a82f624-b48c-434f-bd7c-9fc1e742b8af\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" Jul 06 20:09:38.108555 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.108454 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a82f624-b48c-434f-bd7c-9fc1e742b8af-proc\") pod \"perf-node-gather-daemonset-mgkmm\" (UID: \"1a82f624-b48c-434f-bd7c-9fc1e742b8af\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" Jul 06 20:09:38.108555 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.108489 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a82f624-b48c-434f-bd7c-9fc1e742b8af-lib-modules\") pod \"perf-node-gather-daemonset-mgkmm\" (UID: \"1a82f624-b48c-434f-bd7c-9fc1e742b8af\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" Jul 06 20:09:38.108724 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.108554 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzj9d\" (UniqueName: \"kubernetes.io/projected/1a82f624-b48c-434f-bd7c-9fc1e742b8af-kube-api-access-hzj9d\") pod \"perf-node-gather-daemonset-mgkmm\" (UID: \"1a82f624-b48c-434f-bd7c-9fc1e742b8af\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" Jul 06 20:09:38.210020 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.209922 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzj9d\" (UniqueName: \"kubernetes.io/projected/1a82f624-b48c-434f-bd7c-9fc1e742b8af-kube-api-access-hzj9d\") pod \"perf-node-gather-daemonset-mgkmm\" (UID: \"1a82f624-b48c-434f-bd7c-9fc1e742b8af\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" Jul 06 20:09:38.210020 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.210004 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a82f624-b48c-434f-bd7c-9fc1e742b8af-podres\") pod \"perf-node-gather-daemonset-mgkmm\" (UID: \"1a82f624-b48c-434f-bd7c-9fc1e742b8af\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" Jul 06 20:09:38.210246 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.210046 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a82f624-b48c-434f-bd7c-9fc1e742b8af-sys\") pod \"perf-node-gather-daemonset-mgkmm\" (UID: \"1a82f624-b48c-434f-bd7c-9fc1e742b8af\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" Jul 06 20:09:38.210246 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.210088 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a82f624-b48c-434f-bd7c-9fc1e742b8af-proc\") pod \"perf-node-gather-daemonset-mgkmm\" (UID: \"1a82f624-b48c-434f-bd7c-9fc1e742b8af\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" Jul 06 20:09:38.210246 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.210111 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a82f624-b48c-434f-bd7c-9fc1e742b8af-lib-modules\") pod \"perf-node-gather-daemonset-mgkmm\" (UID: \"1a82f624-b48c-434f-bd7c-9fc1e742b8af\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" Jul 06 20:09:38.210246 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.210195 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a82f624-b48c-434f-bd7c-9fc1e742b8af-podres\") pod \"perf-node-gather-daemonset-mgkmm\" (UID: \"1a82f624-b48c-434f-bd7c-9fc1e742b8af\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" Jul 06 20:09:38.210246 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.210215 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a82f624-b48c-434f-bd7c-9fc1e742b8af-sys\") pod \"perf-node-gather-daemonset-mgkmm\" (UID: \"1a82f624-b48c-434f-bd7c-9fc1e742b8af\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" Jul 06 20:09:38.210246 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.210234 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a82f624-b48c-434f-bd7c-9fc1e742b8af-lib-modules\") pod \"perf-node-gather-daemonset-mgkmm\" (UID: \"1a82f624-b48c-434f-bd7c-9fc1e742b8af\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" Jul 06 20:09:38.210246 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.210215 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a82f624-b48c-434f-bd7c-9fc1e742b8af-proc\") pod \"perf-node-gather-daemonset-mgkmm\" (UID: \"1a82f624-b48c-434f-bd7c-9fc1e742b8af\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" Jul 06 20:09:38.222182 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.222144 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzj9d\" (UniqueName: \"kubernetes.io/projected/1a82f624-b48c-434f-bd7c-9fc1e742b8af-kube-api-access-hzj9d\") pod \"perf-node-gather-daemonset-mgkmm\" (UID: \"1a82f624-b48c-434f-bd7c-9fc1e742b8af\") " pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" Jul 06 20:09:38.239202 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.239174 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" Jul 06 20:09:38.415660 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.415630 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm"] Jul 06 20:09:38.418188 ip-10-0-140-94 kubenswrapper[2578]: W0706 20:09:38.418158 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1a82f624_b48c_434f_bd7c_9fc1e742b8af.slice/crio-2c8ff80dbe9e4f6967ef43ff57d7fc653f918d77e5b309c517bb72ba5a122171 WatchSource:0}: Error finding container 2c8ff80dbe9e4f6967ef43ff57d7fc653f918d77e5b309c517bb72ba5a122171: Status 404 returned error can't find the container with id 2c8ff80dbe9e4f6967ef43ff57d7fc653f918d77e5b309c517bb72ba5a122171 Jul 06 20:09:38.794625 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.794591 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" event={"ID":"1a82f624-b48c-434f-bd7c-9fc1e742b8af","Type":"ContainerStarted","Data":"089bc4c3fd21aad6e8501de6448f13d69c9f8e6a3ab6f2ef9afa1c5408bcd8fd"} Jul 06 20:09:38.795055 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.794644 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" event={"ID":"1a82f624-b48c-434f-bd7c-9fc1e742b8af","Type":"ContainerStarted","Data":"2c8ff80dbe9e4f6967ef43ff57d7fc653f918d77e5b309c517bb72ba5a122171"} Jul 06 20:09:38.795055 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.794727 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" Jul 06 20:09:38.822137 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:38.822077 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" podStartSLOduration=1.822055997 podStartE2EDuration="1.822055997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:09:38.82189848 +0000 UTC m=+2823.450852447" watchObservedRunningTime="2026-07-06 20:09:38.822055997 +0000 UTC m=+2823.451009964" Jul 06 20:09:39.351905 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:39.351878 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-mjdwf_5bdf0c25-68fd-43aa-b54b-baf04fdfe2a0/networking-console-plugin/0.log" Jul 06 20:09:39.936818 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:39.936788 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/1.log" Jul 06 20:09:39.941767 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:39.941747 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-62lvt_09143401-93b0-453e-a738-9b8fa5457a81/console-operator/2.log" Jul 06 20:09:40.538987 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:40.538943 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-cg86p_1bb7129a-88cf-4c6e-b47b-9315e0721dce/download-server/0.log" Jul 06 20:09:41.225532 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:41.225502 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-dqgwt_f9180d35-c91f-42b4-b49d-eebdf8b3a1c3/volume-data-source-validator/0.log" Jul 06 20:09:42.417238 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:42.417207 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8j849_670f9fdc-7f0b-44f5-9277-c88477bde835/dns/0.log" Jul 06 20:09:42.463628 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:42.463587 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8j849_670f9fdc-7f0b-44f5-9277-c88477bde835/kube-rbac-proxy/0.log" Jul 06 20:09:42.701638 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:42.701563 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tnbvj_8c3ea532-8f74-47ba-bd67-89fc5cd148c3/dns-node-resolver/0.log" Jul 06 20:09:43.453290 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:43.453248 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qfm4f_ecbc6515-e4c1-489d-ac42-dd3a967b782c/node-ca/0.log" Jul 06 20:09:44.662818 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:44.662786 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6786q4mq_373da31a-f406-40cb-b233-e4e7f2cbf777/istio-proxy/0.log" Jul 06 20:09:44.811129 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:44.811090 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tfs5f/perf-node-gather-daemonset-mgkmm" Jul 06 20:09:44.878833 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:44.878807 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67db589d55-p5z7q_2342b288-6681-49f1-85ca-261c3fe2e33d/kube-auth-proxy/0.log" Jul 06 20:09:44.961725 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:44.961640 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nxb4p_639168ba-36ec-47a3-a023-6135f27dd8d9/istio-proxy/0.log" Jul 06 20:09:44.995132 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:44.995106 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nxb4p_639168ba-36ec-47a3-a023-6135f27dd8d9/istio-proxy/1.log" Jul 06 20:09:45.095745 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:45.095711 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-45k9w_2e41d8b1-2901-40df-bcc1-9314078eebae/payload-processing/0.log" Jul 06 20:09:45.131440 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:45.131413 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68d9d48ccc-kk7pz_cc56f897-5d59-4451-a517-5bfea063ce0c/router/0.log" Jul 06 20:09:46.264347 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:46.264322 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dk4kn_07c083da-b946-4438-afa0-3c5be0f77b9b/serve-healthcheck-canary/0.log" Jul 06 20:09:47.179648 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:47.179619 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-xgcrt_7bf04227-6bc5-4797-a189-f1c12841bfc4/insights-operator/0.log" Jul 06 20:09:47.180396 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:47.180376 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-xgcrt_7bf04227-6bc5-4797-a189-f1c12841bfc4/insights-operator/1.log" Jul 06 20:09:47.524087 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:47.524057 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cxsxv_61241f90-6bd8-4a8c-8784-02757d7fcb36/kube-rbac-proxy/0.log" Jul 06 20:09:47.631029 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:47.631002 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cxsxv_61241f90-6bd8-4a8c-8784-02757d7fcb36/exporter/0.log" Jul 06 20:09:47.688362 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:47.688332 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cxsxv_61241f90-6bd8-4a8c-8784-02757d7fcb36/extractor/0.log" Jul 06 20:09:49.533197 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:49.533166 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6786q4mq_373da31a-f406-40cb-b233-e4e7f2cbf777/istio-proxy/0.log" Jul 06 20:09:49.713534 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:49.713508 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67db589d55-p5z7q_2342b288-6681-49f1-85ca-261c3fe2e33d/kube-auth-proxy/0.log" Jul 06 20:09:49.753995 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:49.753967 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nxb4p_639168ba-36ec-47a3-a023-6135f27dd8d9/istio-proxy/0.log" Jul 06 20:09:49.789939 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:49.789852 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nxb4p_639168ba-36ec-47a3-a023-6135f27dd8d9/istio-proxy/1.log" Jul 06 20:09:49.868420 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:49.868393 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-45k9w_2e41d8b1-2901-40df-bcc1-9314078eebae/payload-processing/0.log" Jul 06 20:09:49.885061 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:49.885031 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68d9d48ccc-kk7pz_cc56f897-5d59-4451-a517-5bfea063ce0c/router/0.log" Jul 06 20:09:50.296668 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:50.296637 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pv5d5_40393be6-bd01-4617-ace4-5cbdfcd186fd/manager/3.log" Jul 06 20:09:50.296871 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:50.296837 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pv5d5_40393be6-bd01-4617-ace4-5cbdfcd186fd/manager/2.log" Jul 06 20:09:50.327902 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:50.327860 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hrvh5_f55184b1-bd4b-4da5-9f7a-d6099ed678ab/manager/2.log" Jul 06 20:09:50.409502 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:50.409470 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hrvh5_f55184b1-bd4b-4da5-9f7a-d6099ed678ab/manager/3.log" Jul 06 20:09:50.502849 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:50.502816 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722785-sbhn6_32fd3226-5ae9-4c8a-b8a7-6fec412df49c/cleanup/0.log" Jul 06 20:09:50.792432 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:50.792385 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-kkjcw_85caf392-2962-4d96-8ae0-59dbf3366abe/manager/3.log" Jul 06 20:09:50.809996 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:50.809973 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-kkjcw_85caf392-2962-4d96-8ae0-59dbf3366abe/manager/4.log" Jul 06 20:09:50.856409 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:50.856377 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7865995f8c-mwbc8_f1f2d36b-4239-45fa-a6d7-85c316bd426e/manager/2.log" Jul 06 20:09:50.857675 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:50.857656 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7865995f8c-mwbc8_f1f2d36b-4239-45fa-a6d7-85c316bd426e/manager/1.log" Jul 06 20:09:50.882441 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:50.882421 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7865995f8c-mwbc8_f1f2d36b-4239-45fa-a6d7-85c316bd426e/copy-manifests/0.log" Jul 06 20:09:51.066187 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:51.066099 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6786q4mq_373da31a-f406-40cb-b233-e4e7f2cbf777/istio-proxy/0.log" Jul 06 20:09:51.117354 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:51.117323 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-xc6pb_c9acfc53-44f0-4ccd-9768-7d17fb00394d/postgres/0.log" Jul 06 20:09:51.265285 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:51.265257 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67db589d55-p5z7q_2342b288-6681-49f1-85ca-261c3fe2e33d/kube-auth-proxy/0.log" Jul 06 20:09:51.332688 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:51.332603 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nxb4p_639168ba-36ec-47a3-a023-6135f27dd8d9/istio-proxy/0.log" Jul 06 20:09:51.363336 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:51.363306 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nxb4p_639168ba-36ec-47a3-a023-6135f27dd8d9/istio-proxy/1.log" Jul 06 20:09:51.439128 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:51.439096 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-45k9w_2e41d8b1-2901-40df-bcc1-9314078eebae/payload-processing/0.log" Jul 06 20:09:51.461789 ip-10-0-140-94 kubenswrapper[2578]: I0706 20:09:51.461758 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68d9d48ccc-kk7pz_cc56f897-5d59-4451-a517-5bfea063ce0c/router/0.log"